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Foreword 


The  Air  Force  must  improve  its  test  and  evaluation  process  to  correct  the  existing 
problems  and  failures  in  acquiring  electronic  combat  (EC)  systems.  We  must  become 
a  great  deal  better  at  implementing  a  teat  process  which  provides  evidence  that  the 
system  performs  as  expected  in  an  operational  environment  before  proceeding  to 
full-scale  production.  Operational  test  and  evaluation  (OT&E)  is  the  principal  means 
for  assessing  the  system’s  operational  effectiveness  and  suitability  and  determining 
the  extent  that  the  EC  system  satisfies  operational  requirements. 

Results  from  OT&E  are  of  particular  value  to  decision  makers  because  they  are 
derived  from  test  agencies  that  are  independent  of  the  developing  and  using  com¬ 
mands.  They  provide  information  that  decision  makers  can  use  when  determining 
the  merits  of  continuing  the  acquisition  program  and  reducing  program  risks.  And 
yet,  when  operational  testing  does  not  provide  sufficient  information  or  the  type  of 
information  decision  makers  are  looking  for  at  program  decision  points,  the  acquisi¬ 
tion  program  becomes  stalled  until  further  testing  is  completed,  adding  time  and 
expense  to  the  program. 

Maj  Frederick  L.  Wright  has  taken  a  significant  step  toward  addressing  the  con¬ 
cern  that  the  Air  Force  does  not  have  a  structured  EC  test  process.  He  has  designed 
a  process  that  can  standardize  the  test  procedures  for  the  evaluation  of  EC  systems 
and  supply  the  type  of  information  decision  makers  are  expecting  at  the  conclusion  of 
testing.  His  study  offers  many  insights  into  the  chaUenge  of  testing  EC  systems  and 
provides  the  reader  with  a  description  of  the  type  of  OT&E  to  be  conducted  in  each 
phase  of  the  acquisition  process.  I  am  convinced  that  anyone  associated  with  OT&E 
of  EC  systems  or  any  other  weapon  system  would  benefit  greatly  from  a  close  ex¬ 
amination  of  this  study. 


DONALD  M.  DOUGLAS,  Col,  USAF 
Director  of  System  Anal3rsis 
Headquarters  Air  Force  Operational 
Test  and  Evaluation  Center 
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Preface 


In  the  mid-  to  late  19808  there  was  a  proliferation  of  new  threat  systems  being 
produced  and  deployed.  This  expansion  weakened  the  Air  Force  s  ability  to  success¬ 
fully  operate  and  support  a  weapon  system  in  its  intended  operational  environment. 
As  a  result,  various  new  electrojaic  combat  (EC)  system  acquisition  programs  were 
started  in  addition  to  upgrades  to  existing  EC  systems.  They  were  directed  toward 
correcting  deficiencies  in  our  current  capabilities  to  identify  and  counter  the  various 
threat  S3r8tems. 

Initially,  EC  systems  were  developed  under  a  “quick-reaction”  program  that  at¬ 
tempted  to  satisfy  the  immediate  need  of  countering  the  enemy’s  threat  system. 
Consequently,  the  test  and  evaluation  (T&E)  that  took  place  ended  up  being  more  of 
a  trial-and-error  process.  In  recent  years  it  has  become  apparent  that  these  systems 
did  not  always  work  as  desired  despite  the  expense  of  developing  and  deploying 
them.  This  has  led  various  levels  of  government  to  question  why  we  can’t  produce  an 
EC  system  that  does  the  job  as  intended  without  requiring  huge  sums  of  money  and 
an  inordinate  number  of  years  to  develop.  This  t3rpe  of  question  and  increased 
congressional  scrutiny  of  EC  system  acquisition  programs  led  to  a  broad  area  review 
of  the  EC  acquisition  process. 

One  of  the  fmdings  from  the  review  hig^ighted  the  fact  that  there  is  no  stan¬ 
dardized  test  process  for  EC  systems  like  there  is  for  aircraft.  As  a  result,  decision 
makers  are  not  receiving  the  kind  of  information  they  need  in  making  acquisition 
decisions.  In  addition,  the  Department  of  Defense  (DOD)  inspector  general  (IG) 
reviewed  several  operational  test  and  evaluation  (OT&E)  programs  and  concluded 
that  OT&E  would  have  more  impact  on  acquisition  decisions  if  it  did  not  get  cau^t 
up  in  the  “test-fix-test  scenario”  that  began  in  developmental  testing.  As  a  conse¬ 
quence,  the  production  of  meaningful  test  results  used  by  the  decision  makers  to 
assess  an  EC  system’s  operational  effectiveness  and  suitability  has  not  been  com¬ 
pletely  satisfied. 

Furthermore,  there  is  a  growing  perception  that  OT&E  has  become  an  extension  of 
developmental  testing  and  cannot  produce  meaningful  acquisition  information  for 
the  decision  makers.  This  is  due  in  part  to  the  limitations  and  challenges  of  evaluat¬ 
ing  the  EC  system’s  contribution  to  the  overall  success  of  the  mission.  Many  of  these 
limitations  are  caused  by  an  inadequate  operational  test  environment  that  is  not 
representative  of  the  actual  threat  environment.  A  test  meth  xi  must  be  developed  to 
evaluate  EC  systems  in  a  test  environment  that  faithfully  represents  the  actual 
operational  environment.  Additionally,  the  operational  test  agency  (OTA)  gets 
cau^t  up  in  assessing  or  evaluating  the  performance  of  the  EC  system  at  a  system 
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level  instead  of  at  the  mission  level.  Decision  makers  want  to  know  how  effective  the 
EC  system  is  in  contributing  to  the  success  of  the  mission.  The  OTAs  must  do  a 
better  job  in  establishing  mission-level  measures  that  reflect  the  EC  system’s  con¬ 
tribution  to  the  mission.  With  the  proper  apphcation  of  the  T&E  tools,  decision 
makers  can  get  meaningful  test  results  that  will  show  the  EC  system’s  effectiveness 
in  supporting  the  mission  objectives. 

This  study  analyzes  the  flndings  from  the  broad  area  and  DOD  IG’s  reviews,  and  it 
devises  an  EC  test  process  that  incorporates  the  principles  of  a  scientific  test 
methodology.  Employing  the  methods  associated  with  a  scientific  test  process  adds 
discipline  and  structure  to  the  evaluation  of  EC  systems  with  the  intent  cf  providing 
meaningful  information  to  the  decision  makers.  This  is  directed  to  those  OTAs  and 
individuals  who  are  tasked  to  conduct  OT&E  of  EC  systems.  It  begins  by  reviewing 
the  t3rpes  of  T&E  and  the  different  stages  of  operational  testing,  then  many  of  the 
hmitations  and  challenges  to  EC  system  testing  are  reported  to  call  attention  to  the 
difflculties  in  testing  EC  systems. 

To  determine  whether  the  EC  system  can  indeed  identify  or  counter  the  threat  will 
require  the  generation  of  sufficient  test  data.  This  study  describes  the  type  of  T&E 
tools  needed  to  generate  the  test  data  as  well  as  their  purpose  in  supporting  the  EC 
test  process.  Following  the  description  of  the  T&E  tools  are  six  steps  that  form  the 
foundation  for  a  scientific  test  process.  The  scientific  test  process  can  be  applied  to 
any  stage  of  OT&E  while  incorporating  the  discipline  and  structure  needed  to 
evaluate  the  operational  effectiveness  and  suitability  test  objectives. 

Finally,  it  is  important  to  understand  the  tasks  *hat  are  accomplished  in  each 
phase  of  the  acquisition  process  and  how  the  EC  test  process  supports  the  decision 
makers.  The  five  phases  of  the  acquisition  process  provide  an  excellent  means  to 
implement  the  EC  test  process.  This  study  delineates  each  phase  along  with  the 
responsibilities  of  the  system  program  office  and  the  OTA  in  supporting  the  acquisi¬ 
tion  of  EC  systems.  In  addition,  it  points  out  where  and  how  the  T&E  tools  can  lend 
assistance  to  the  operational  assessment  or  evaluation  in  each  phase  of  the  acquisi¬ 
tion  process. 

Implementing  the  recommendations  in  this  study  will  structure  an  EC  test  process 
that  will  provide  the  discipline  needed  to  overcome  the  limitations  and  challenges 
associated  with  testing  EC  systems.  By  using  T&E  tools  properly  and  applying  the 
methods  related  to  the  scientific  test  process,  we  can  generate  a  reliable  estimate  of 
the  operational  effectiveness  and  suitability  of  EC  systems  at  the  mission  level.  'The 
scientific  test  process  will  also  help  the  OTAs  avoid  the  test-fix-test  scenario  that 
began  in  developmental  testing  by  preparing  the  tester  for  the  evaluation  and  by 
reporting  the  results  from  the  evaluation  at  the  conclusion  of  the  test  process.  Fur¬ 
thermore,  the  test  process  described  in  this  study  gives  the  decision  makers  meaning¬ 
ful  information  on  which  they  can  base  their  acquisition  decisions  and  the  confidence 
that  the  test  results  reflect  the  way  the  system  will  work  if  deployed.  Adopting  the 


recommendations  in  this  study  will  provide  a  cost-effective  and  efficient  method  to 
test  EC  systems.  The  outcome  wU  be  an  EC  system  that  will  function  in  its  intended 
operational  environment  and  contribute  to  the  overall  success  of  the  mission. 

/ 

FREDERICK  L,  WRIGHT.  Maj,  USAF  • 

Research  Fellow 
Airpower  Reseaich  Institute 


XV 


Acknowledgments 


I  would  like  to  acknowledge  several  people  without  whose  support  and  help  I 
would  never  have  gotten  off  the  ground  with  this  study.  I  want  to  thank  Maj  Gen 
Peter  D.  Robinson,  former  commander  of  AFOTEC,  for  selecting  me  to  do  the  re¬ 
search  into  OT&E  of  EC  systems.  General  Robinson  understands  the  importance  of 
OTftE  and  has  recognized  the  fact  that  the  current  test  process  has  not  satisfied  the 
needs  of  the  decision  authorities.  I  would  like  to  thank  Col  Mike  Douglas  for  his 
confidence  and  support  during  my  time  at  Maxwell  AFB,  Alabama.  My  thanks  also 
go  to  Mr  Mike  StoUe  for  reviewing  the  draft  and  providing  comments  to  keep  me  on 
track  as  OT&E  evolves  to  meet  the  needs  of  the  acquisition  process. 

I  want  to  especially  thank  Lt  Col  Mike  Reich  for  the  many  discussions  we  had  on 
EC  system  testing.  His  experience  and  insight  into  T&E  of  EC  systems  have  been 
invaluable  in  helping  me  to  eliminate  inconsistencies  in  my  study.  I  would  like  to 
thank  Lt  Col  Roy  Williams  for  the  support  he  has  given  me  in  reviewing  various 
portions  of  the  document,  especially  in  the  area  of  T&E  tools. 

I  eun  particularly  grateful  to  my  reading  group  chairman.  Dr  Lewis  B.  Ware,  for 
getting  me  started  and  teaching  me  how  to  go  about  writing  a  research  report.  His 
suggestions  improved  both  the  organization  and  tone  of  the  report  while  strengthen¬ 
ing  the  importance  of  the  EC  test  process.  A  special  thanks  to  my  editor,  Ms  Marion 
S.  Gorrie,  who  enhanced  the  quality  and  readability  of  the  study.  I  am  most  grateful 
for  her  fine  work. 

Most  importantly,  !  want  to  express  my  sincere  appreciation  to  my  wife,  Elizabeth, 
and  daughter,  Kaitlin,  for  their  love  and  understanding  diiring  those  times  when  1 
was  struggling  with  this  study.  Tlieir  presence  was  very  important  to  me  and  made 
all  the  difference  in  ensuring  my  success  in  completing  this  study. 


Executive  Summary 


The  current  electronic  combat  (EC)  test  process  lacks  discipline  smd  structure 
because  there  is  no  standardized  operational  test  process  used  in  t  valuating  EC 
systems.  The  purpose  of  this  study  is  to  describe  a  test  process  that  incorporates 
discipline  and  structure  into  the  operational  test  and  evaluation  (OT&E)  of  EC  sys¬ 
tems  with  the  intent  of  providing  seeded  information  to  the  decision  makers. 

The  failure  of  the  test  process  for  EC  systems  can  be  traced  to  several  limitations 
and  challenges.  One  of  the  limitations  has  to  do  with  an  inadequate  field  test  en¬ 
vironment  that  does  not  truly  represent  the  operational  environment.  Additionally, 
the  operational  test  agencies  (OTA)  face  the  challenge  of  establishing  mission-level 
measures  amd  coming  up  with  evaluation  criteria  for  those  measures  prior  to  the 
start  of  OT&E.  Another  limitation  to  the  EC  test  process  is  the  absence  of  complete 
intelligence  on  the  threat  systems.  Finally,  the  OTAs  do  not  have  a  method  that 
reports  the  contribution  toward  mission  success  from  an  EC  system  that  is  in¬ 
tegrated  with  and  hi^ily  dependent  on  other  on-board  avionics.  A  test  process  must 
be  developed  that  addresses  these  limitations  and  challenges. 

In  this  study  the  author  resolves  these  challenges  by  applying  the  proper  mix  of 
test  and  evaluation  tools  and  by  utilizing  the  methods  associated  with  a  scientific 
test  process.  The  ri^t  mix  and  application  of  test  tools  can  provide  the  decision 
makers  with  information  regarding  the  EC  system’s  effectiveness  and  suitability  in 
an  operationally  representative  environment  at  a  mission  level.  This  is  accomplished 
by  applying  a  scientific  test  methodology  that  provides  the  discipline  and  structure  to 
the  EC  test  process. 

On  the  basis  of  the  above,  the  author  recommends  that  the  OTAs  (1)  institute  the 
scientific  test  process,  (2)  educate  and  train  the  appropriate  personnel,  and  (3)  clarify 
the  requirements  in  AFU  55-43,  Management  of  Operational  Test  and  Evaluation. 

This  study  is  targeted  to  the  OTAs,  test  teams,  test  managers,  and  members  of  the 
test  support  group  who  have  been  tasked  to  conduct  OT&E  of  EC  systems.  It  can  be 
both  a  primer  for  someone  new  to  the  operational  evaluation  of  EC  systems  or  an 
essay  for  the  professional  tester.  It  provides  a  basic  description  of  the  T&E  tools 
used  to  evaluate  EC  systems  as  well  as  a  scientific  test  process  that  will  add  dis¬ 
cipline  and  structure  to  the  test  process.  In  addition,  this  study  gives  the  operational 
test  manager  and  test  teams  some  insight  into  the  acquisition  process,  the  respon¬ 
sibilities  of  those  involved  in  testing  EC  systems,  and  the  kind  of  information  the 
decision  makers  are  looking  for  at  each  milestone  decision  point. 


Chapter  1 


Introduction  to  Operational 
Test  and  Evaluation 


Operational  teat  and  evaluation  (OT&E)  ia  an  important  part  of  the  acquiai- 
tion  proceaa.  OT&E  ia  teating  conducted  in  aa  reahatk  an  operational  en¬ 
vironment  aa  practical  to  evaluate  operational  effectiveneaa  and  auitability 
objectivea  and  to  provide  an  eatimate  of  the  syatem’a  military  utility.  OT&E 
providea  credible  information  to  the  Air  Force  and  Department  of  Defenae 
(DOD)  decision  makers  who  rely  on  OT&E  findings  at  various  program  mile¬ 
stones.  The  findings  contribute  to  decisions  on  the  acquisition  of  new  sys¬ 
tems,  or  upgrades  to  existing  ^tems,  and  to  the  status  of  their  operational 
capabilities. 

Testing  provides  the  basis  for  an  evaluation  by  obtaining,  verifying,  or 
producing  data.^  Ihe  evaluation  furnishes  a  method  to  judge  the  S3rstem'8 
achievement  of  required  operational  effectiveness  and  suitability  objectives  by 
analyzing  quantitative  and  qualitative  data  derived  from  the  test.  The  pur¬ 
pose  of  testing  electronic  combat  (EC)  systems  is  to  see  if  they  meet  the  using 
command’s  specific  mission  need  or  satisfy  a  deficiency  in  terms  of  critical 
operational  issues. 

As  a  result  of  OT&E  identifying  operational  inadequacies  in  several  EC 
systems,  the  Air  Force  initiated  in  October  1988  a  broad  area  review  to  deter¬ 
mine  if  there  were  deficiencies  in  the  EC  S3r8tem  acquisition  process.  The 
review  concluded  that  EC  testing  lacks  the  discipline  and  essential  elements 
of  a  scientific  approach.^  Because  of  the  lack  of  discipline  and  structure  in  the 
test  process,  a  perception  has  been  created  that  the  operational  testing  com¬ 
munity  cannot  produce  meaningful  information  for  the  decision  makers. 
Another  perception  is  that  OT&E  does  not  have  the  impact  on  the  acquisition 
milestone  decisions  as  envisioned  by  Congress. 

The  purpose  of  this  study  is  to  give  the  operational  test  agencies  (OTA)  a 
standardized  test  process  t^t  includes  the  discipline  and  structure  of  a  scien¬ 
tific  test  approach  and  that  can  be  applied  to  the  OT&E  of  EC  sjrstems.  The 
stu<fy  describes  the  test  and  evaluation  (T&E)  tools  that  are  available  to  the 
test  manager  and  their  application  in  the  EC  test  process.  It  outlines  the 
procedures  for  conducting  an  orderly  scientific  test  process  needed  to  test  EC 
S3r8tems.  The  scientific  test  process  will  help  overcome  some  of  the  limitations 
and  challenges  associated  with  the  current  EC  test  process.  The  study  con¬ 
cludes  by  providing  the  test  manager  with  a  description  of  the  information 
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aiTorded  at  each  milestone  point  and  how  OT&E  is  used  to  support  the  ac¬ 
quisition  process. 

Having  begun  by  defining  the  purpose  of  OT&E,  this  chapter  defines  the 
different  stages  of  OT&E  and  where  they  fit  into  the  framework  of  the  ac¬ 
quisition  process  ifig.  1).  Then  the  chapter  discusses  several  limitations  and 
challenges  to  effective  OT&E  along  with  two  examples  of  where  the  current 
EC  test  process  has  broken  down.  Finally,  the  chapter  concludes  with  state¬ 
ments  made  by  the  chief  of  staff  of  the  Air  Force  (CSAF)  on  the  EC  acquisition 
process  and  points  out  several  findings  from  an  Air  Force  ad  hoc  group  study 
on  the  EC  test  process. 

Types  of  Test  and  Evaluation 

Testing  is  normally  divided  between  developmental  and  operational  events. 
Developmental  test  and  evaluation  (DT&E)  and  OT&E  can  occur  in  any  phase 
of  the  acquisition  process.  Usually  OT&E  follows  DT&E,  but  they  may  over¬ 
lap  or  be  combined.  DT&E  involves  an  engineering  analysis  of  the  system  to 
ensure  that  it  meets  the  design  specifications  and  performance  requirements 
and  that  it  accomplishes  the  developmental  objectives.  DT&E  is  usually  con¬ 
ducted  by  the  developing  contractor  or  system  program  office  (SPO)  in  some 
type  of  controlled  laboratory  or  field  test  environment  with  a  single  protot5rpe 
system.  Generally  DT&E  does  not  go  much  beyond  testing  an  individual 
prototype  system  against  a  single  threat  system  (one-versus-one),  or  at  most 
in  a  one-versus-few  test  environment.  As  a  result,  the  system’s  demonstrated 
performance  at  the  end  of  developmental  testing  may  not  be  truly  repre¬ 
sentative  of  how  it  will  perform  in  an  operational  environment. 

Through  OT&E,  the  OTAs  wall  determine  if  the  EC  system’s  performance 
meets  the  user’s  operational  requirements  for  combat  or  other  mission  needs. 
The  Air  Force’s  OTAs  are  tasked  through  the  program  management  directive 
(PMD)  to  manage  the  independent  OT&E  of  the  electronic  combat  system. 

Five  major  acquisition  phases  and  milestone  decision  points  (see  fig.  1) 
provide  a  developmental  framework  to  implement  a  process  to  ensure  the  EC 
system  meets  the  user’s  operational  requirements.  Except  for  the  beginning 
milestone,  each  milestone  decision  is  supported  by  some  t3rpe  of  operational 
assessment  or  evaluation.  Decision  makers  rely  on  these  assessments  and 
evaluations  to  make  knowledgeable  decisions  during  each  phase  of  the  ac¬ 
quisition  process.  Furthermore,  results  from  OT&E  provide  the  confidence 
decision  makers  need  in  deciding  whether  the  program  should  enter  the  next 
phase  of  the  acquisition  process. 

Operational  assessments  and  evaluations  support  the  milestone  decision 
points  wnth  information  drawm  from  various  sources.  Depending  on  which 
acquisition  phase  the  program  is  in  and  the  nature  of  the  test,  these  sources  of 
information  can  include  observing  advanced  developmental  prototypes, 
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models,  and  simulations;  using  data  obtained  from  modeling  and  simulation 
or  from  combined  DT&E/OT&E  testing;  or  anal3rzing  data  from  dedicated 
OT&E.  Whatever  the  source,  the  OTA  must  make  sure  that  its  conclusions 
truly  report  the  capabiHties  of  the  system  in  terms  of  operational  require¬ 
ments. 

An  important  part  of  the  decision-making  process  for  all  acquisition 
prc^ams  is  the  cost/operational  effectiveness  analysis  (COEA)  process.  Air 
Force  Regulation  (APR)  57-1,  Air  Force  Mission  Needs  and  Operational  Re¬ 
quirements  Process,  states  that 

the  COEA  allows  the  MAJCOM/CC  to  make  comparisons  of  alternative  solutions  on 
the  basis  of  cost  and  operational  effectiveness,  documents  analytical  rationale  for 
preferring  one  alternative  over  another,  helps  to  justify  the  need  for  starting  or 
continuing  an  acquisition  program,  and  effectively  communicates  the  decision  at  all 
levels  in  operations,  cost,  and  acquisition  communities.’ 

The  major  command  (MAJCOM)  responsible  for  the  mission  area  that  in¬ 
cludes  the  identified  deficiency  or  need  will  conduct  concept  studies,  identify 
and  evaluate  potential  alternative  solutions,  and  prepare  the  COEA.  The 
implementing,  supporting,  and  participating  commands  and  agencies  will  pro¬ 
vide  support  to  the  MAJCOM  in  the  COEA  process  as  directed  in  the  program 
management  directive.  The  MAJCOM  prepares  a  COEIA  for  the  milestone  I 
and  II  decision  points  and  updates  the  COEA  for  the  other  milestones  as 
required. 

The  COEIA  is  intended  to  aid  decision  makers  in  judging  whether  the  alter¬ 
natives  offer  sufficient  military  benefit  to  be  worth  the  cost.  The  process  will 
involve  decision  makers  and  staffs  at  all  levels  in  the  discussion  of  reasonable 
alternatives.  The  COEA  process  will  also  document  the  acquisition  decisions 
by  providing  a  record  of  the  alternatives  considered  at  each  milestone  decision 
point.  A  comprehensive  T&E  program  places  confidence  in  the  key  assump¬ 
tions  and  estimates  that  go  into  the  COEA.  Results  from  testing  then  provide 
needed  information  for  the  COEA  process  as  well  as  analyzing  the  operational 
effectiveness  and  suitability  of  the  system. 

In  an  effort  to  help  reduce  risk  in  early  acquisition  decisions,  the  DOD  has 
instructed  the  OTAs  to  examine  systems  in  the  early  stages  of  the  acquisition 
process  before  there  are  any  production-representative  systems  to  test.  The 
General  Accounting  Office  (GAO)  agrees  Mrith  this  philosophy  and  cites  the 
following  in  its  report  to  the  chairman  of  the  Senate  Committee  on 
Governmental  Affairs; 

When  OT&E  is  done  before  initial  production  (approval],  information  is  available  on 
potential  shortcomings  that  would  not  be  foreseen  through  developmental  testing. 
Further,  OT&E  results  permit  decision  makers  to  assess  whether  potentially  costly 
modiflcations  are  needed.^ 

Therefore,  early  involvement  by  the  OTAs  can  provide  decision  makers  with 
additional  knowledge  to  manage  risk  with  key  indicators  provided  through 
operational  assessments  and  evaluations  at  each  milestone. 


4 


If  an  OTA  is  involved  in  the  concept  exploration  and  definition  phase  to 
support  a  milestone  I  decision  or  during  the  demonstration  and  validation 
phase  to  support  a  milestone  II  decision,  it  is  called  an  early  operational 
assessment  (EOA)  (see  fig.  1).  When  test  activity  supports  a  low  rate  initial 
production  (LRIP)  or  similar  decision  before  milestone  III,  it  is  known  as  an 
operational  assessment  (OA).  EOAs  and  OAs  provide  the  information  to  as¬ 
sess  a  system’s  potential  capability  to  meet  user  requirements  and  the 
program’s  progress  toward  producing  an  operationally  effective  and  suitable 
system. 

However,  it  is  of  paramount  importance  that  the  decision  makers  not  judge 
the  performance  of  the  system  on  the  operational  assessment  alone.  While  a 
national  security  fellow  at  Harvard  University,  Col  Robert  Behler  made  a 
similar  comment  in  his  study,  “Defense  Acquisition  in  the  Post  Cold  War 
Era,”  when  describing  the  approach  to  an  EOA. 

It  is  essential  that  the  ar<(Ui8ition  decision  makers  fully  understand  that  an  EOA 
(or  OAI  is  not  a  substitute  for  dedicated  Operational  Test  and  Evaluation,  only  a 
tool  for  making  prudent  lecisions  early  in  the  acquisition  process.  Only  by  gather¬ 
ing  empirical  data  under  realistic  conditions  will  the  actual  operational  capabilities 
of  a  system  be  understood.® 

Operational  assessments  are  generally  intended  to  (1)  review  the  status  of 
program  documentation,  with  emphasis  on  user  requirements  and  testable 
criteria;  (2)  review  teat  planning  issues  dealing  with  the  schedule  and  re¬ 
sources;  (3)  highlight  significant  programmatic  voids  and  trends  that  could 
impact  the  system’s  ability  to  meet  user  requirements;  and  (4)  draw  con¬ 
clusions  from  limited  field  tests  or  simulations  as  directed.^ 

The  purpose  of  the  OA  is  not  to  critique  DT&E  but  to  comment  on  impor¬ 
tant  trends  identified  during  the  assessment  that  could  impact  user  require¬ 
ments  and  to  ensure  that  the  user’s  operational  concerns  are  addressed  in  the 
developmental  process.  An  operational  assessment  provides  a  method  for  the 
operational  test  agency  to  interact  with  the  user  and  developer  early  in  the 
acquisition  process  to  make  sure  operational  requirements  are  clearly  estab¬ 
lished  and  defined  with  meaningful  OT&E  criteria.  OAs  are  based  on  all 
information  relevant  to  the  program,  such  as  data  from  development  testing, 
user  trials,  interim  OT&E  results,  and  modeling/simulation.  For  the  OTAs, 
the  main  thrust  of  both  the  EOA  and  OA  is  to  ensure  that  the  test  program  is 
ready  to  enter  the  next  phase  of  the  OT&E. 

It  is  during  the  engineering  and  manufacturing  development  phase  that  the 
test  program  enters  initial  operational  test  and  evaluation  (lOT&E)  (see  fig. 
1).  lOT&E  begins  as  early  as  possible  in  this  phase  of  the  acquisition  process 
and  initially  may  be  combined  with  developmental  test  and  evaluation  as  they 
share  the  same  prototjrpe  systems.  The  testing  that  is  conducted  on  the 
preproduction  systems  or  prototypes  is  designed  to  provide  an  estimate  of  the 
system’s  operational  effectiveness  and  suitability.  Many  of  the  same  re- 
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sources  and  T&E  tools  used  in  DT&E  may  be  used  in  lOT&E.  lOT&E 
concludes  with  a  dedicated  phase  of  testing  using  production  or 
production-representative  systems  and  is  normally  completed  before  the  first 
major  production  decision  (milestone  III). 

Follow-on  operational  test  and  evaluation  (FOT&E)  is  normally  conducted 
alter  the  fiiU-rate  production  decision  has  been  made.  In  this  final  phase  of 
OT&E,  testing  is  conducted  on  production  systems  in  an  operational  test 
environment.  FOT&E  is  used  to  verify  operational  effectiveness  and 
suitability  results  determined  during  lOT&E,  to  identify  operational  deficien¬ 
cies,  to  evaluate  system  changes,  or  to  reevaluate  the  system  against  chang¬ 
ing  operational  needs. 

The  binding  document  that  is  used  to  plan,  review,  and  approve  the  T&E 
process  is  the  test  and  evaluation  master  plan  (TEMP).  It  supports  the  ac¬ 
quisition  process  by  depicting  the  overall  structure  and  objectives  of  the  test 
program.  AFR  80-14,  Test  and  Evaluation,  states  that  “a  TEMP  is  required 
for  all  HQ  USAF  programs  directed  by  a  program  management  directive 
(PMD).”^  It  identifies  the  critical  issues,  test  objectives,  evaluation  criteria, 
system  characteristics,  responsibilities,  resources,  and  schedules  that  form 
the  framework  for  the  T&E  program.®  A  framework  for  T&E  allows  those 
issues  associated  with  the  test  schedule  and  resource  requirements  to  be 
resolved  and  the  OT&E  test  plan  developed.  Either  the  SPO  or  an  agency 
designated  in  the  PMD  will  prepare  the  TEMP.  The  preparer  will  receive 
assistance  from  the  participating,  operating,  and  supporting  commands  and 
OTAs  in  developing  and  updating  the  TEMP  as  required. 

Each  stage  of  OT&E  serves  a  specific  purpose  in  the  acquisition  process. 
The  earlier  stages  of  OT&E  furnish  the  developer  and  decision  maker  with 
information  on  the  progress  of  the  EC  system  in  meeting  operational  require¬ 
ments.  It  is  also  used  to  prepare  for  and  plan  the  operational  evaluation.  The 
final  stages  of  OT&E  are  designed  to  give  the  user  and  decision  makers  an 
estimate  of  how  well  the  EC  system  met  or  did  not  meet  the  user’s  require¬ 
ments. 

OT&E  has  a  responsibility  in  the  early  phases  of  the  acquisition  process  to 
provide  information  on  the  EC  system’s  potential  in  meeting  the  user’s  mis¬ 
sion  need.  However,  this  creates  quite  a  challenge  to  the  OTAs  because  in 
most  cases  there  are  no  production-representative  systems  available  to  test. 
In  the  later  stages  of  OT&E,  continual  trade-offs  made  in  the  design  and 
development  of  the  EC  system  can  delay  fielding  a  system  that  is  ready  for 
dedicated  operational  testing.  As  a  result,  the  amount  and  extent  of  opera¬ 
tional  testing  that  can  be  accomplished  before  the  scheduled  decision  point 
are  sometimes  affected.  Therefore,  OT&E  has  not  always  been  effective  in 
estimating  the  performance  of  the  EC  system  before  the  initial  production 
decision  is  made.  To  field  an  EC  system  that  meets  the  user’s  operational 
effectiveness  and  suitability  requirements,  these  and  other  challenges  to  the 
EC  test  process  must  be  overcome. 
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Challenges  and  Limitations 


Most  challenges  to  OT&E  come  in  the  form  of  limitations  or  shortfalls  in 
evaluating  an  EC  S3rstem  in  a  field  test  environment  that  represents  a  realis¬ 
tic  mission  scenario.  Historically,  these  challenges  have  influenced  the  EC 
test  programs  in  different  ways,  but  the  end  result  is  always  the  same — the 
perception  that  OT&E  cannot  produce  meaningful  test  results  and  is  nonsup- 
portive  of  congressional  intent.  However,  OT&E  does  identify  inadequacies  in 
the  effectiveness  and  suitability  of  the  EC  S3rstems  tested.  The  problem  is 
with  the  acquisition  process,  or  more  specifically,  with  the  implementation  of 
the  EC  test  process  rather  than  with  OT&E  in  general.  The  General  Account¬ 
ing  Office  cites  the  following  in  its  report  to  the  Senate  Committee  on 
Governmental  Affairs: 

[The]  selection  of  test  sites  [has]  not  always  been  representative  of  operating  en¬ 
vironments,  test  objectives  and  evaluation  criteria  have  not  always  been  estab¬ 
lished,  test  resources  have  not  always  been  available  or  adequate.* 

To  adequately  test  EC  systems,  improvements  must  be  made  in  the  test 
process  to  correct  these  limitations  and  shortfalls.  Correcting  these  inade¬ 
quacies  will  give  the  decision  maker  the  knowledge  to  judge  whether  the 
system  meets  its  operational  effectiveness  and  suitability  performance  re¬ 
quirements. 

Ideally,  OT&E  of  electronic  combat  systems  should  be  conducted  in  an 
unconstrained,  realistic  operational  environment.  This  would  provide  the 
necessary  test  data  to  evaluate  the  effectiveness  of  the  system  in  its  intended 
environment  and  its  suitability  in  the  field.  However,  operational  testing 
usually  ends  in  a  compromise  between  the  ideal  and  the  possible.^*’ 

OTAs  are  faced  with  testing  the  EC  system  in  a  field  environment  that  is 
generally  accepted  by  the  decision  maker  as  the  best  available,  but  still  lacks 
that  sense  of  true  realism  that  the  decision  maker  would  like  to  see.  How¬ 
ever,  to  provide  the  realism  and  threat  density  representative  of  the  opera¬ 
tional  environment  many  changes  and  improvements  are  needed  to  the  field 
test  ranges.  And,  although  the  test  program  is  a  small  part  of  the  total  costs 
in  acquiring  a  sjrstem,  it  still  receives  its  share  of  scrutiny  when  it  comes  to 
allocating  funds.  Since  the  T&E  infrastructure  used  to  support  the  test  pro¬ 
gram  does  not  always  receive  adequate  funding  to  facilitate  needed  improve¬ 
ments,  testing  is  affected  also.  OTAs  must  look  for  ways  to  keep  the  cost  of 
OT&E  within  reason,  and  they  must  accept  the  challenge  of  conducting  realis¬ 
tic  OT&E  under  various  budget  constraints. 

Currently,  the  level  of  realism  needed  in  the  field  cannot  be  achieved  due  to 
the  scarcity  and  quality  of  the  threat  simulators.  The  result  is  a  field  test 
that  evaluates  the  EC  system’s  performance  in  a  one-versus-one  or  at  best 
one-versus-few  scenario.  Furthermore,  because  of  the  ever-increasing  use  of 
the  electromagnetic  spectrum  in  EC,  field  test  ranges  must  provide  a  multi- 
spectral  threat  environment  in  which  to  test  EC  systems.  However,  generat¬ 
ing  a  representative  multispectral  threat  environment  is  difficult  without  a 


substantial  investment  in  test  resources  and  range  improvements.  Even  if 
there  were  unlimited  funds  to  build  and  improve  the  field  test  remges,  it  is 
still  not  possible  to  reproduce  the  operational  scenario  exactly.  You  caimot 
build  an  infrastructure  to  reflect  every  possible  scenario  combination  imd  you 
cannot  allow  the  test  aircraft  to  be  shot  down  by  the  threat  systems.  So,  while 
the  decision  maker  wants/desires  as  much  realism  as  possible,  achieving  it  is 
expensive,  forcing  trade-offs  between  satisfying  the  requirement  for  realistic 
operational  testing  and  the  costs  in  conducting  the  test. 

In  most  cases,  the  OTAs  are  faced  with  the  challenge  of  establishing  opera¬ 
tional  test  measures  and  evaluation  criteria  without  an  adequate  mission 
statement.  It  is  easy  to  obtain  technical  performance  measures  from  contract 
specifications.  But  translating  them  into  mission-level  measures  that 
describe  the  operational  effectiveness  of  the  EC  system  is  a  challenge  that 
must  be  addressed  through  an  improved  EC  test  process. 

Currently,  the  OTAs  end  up  with  measures  of  effectiveness  that  are  very 
similar  to  the  technical  specifications  used  in  developmental  T&E,  such  as 
detection  times  and  ranges,  correct  threat  prioritization,  threat  identification, 
and  threat-processing  capabilities.  Consequently,  the  decision  makers  have  to 
base  their  acquisition  decisions  on  results  from  technical  performance 
measures  instead  of  mission-level  results.  Meeting  specifications  does  not 
guarantee  that  the  S3rstem  will  perform  in  the  operational  environment  as 
envisioned.*^ 

Decision  makers  really  want  to  know  how  the  EC  system  supports  the 
mission  objectives  in  an  operational  environment  and  if  it  performs  satisfac¬ 
torily  in  this  environment.  The  current  EC  test  process  does  not  go  far 
enough  in  answering  these  kinds  of  questions.  An  improved  EC  test  process 
may  never  be  able  to  provide  all  the  answers,  but  with  mission-level  measures 
and  meaningtiil  evaluation  criteria,  OT&E  can  provide  some  determination  of 
how  effective  the  system  is  and  its  suitability  in  an  operational  environment. 

One  of  the  challenges  of  developing  and  testing  EC  systems  is  the  constant¬ 
ly  evolving  threat  and  the  lag  in  gathering  intelligence  on  the  threat  sys¬ 
tems.*^  Developers  and  testers  of  EC  systems  must  have  specific  and  detailed 
intelligence  on  the  threat  systems  they  are  likely  to  encounter  in  the  wartime 
environment.  In  most  cases,  sufficient  intelligence  is  not  available  as  the 
design  of  the  EC  system  proceeds.  New  threat  intelligence  dictates  changes  in 
the  design  of  the  EC  system.  As  a  result,  additional  development  time  is 
needed  to  incorporate  this  information  into  the  system  and  to  verify  that  any 
new  technology  needed  to  support  the  design  changes  will  work.  This  makes 
the  OTA’s  job  of  testing  a  production-representative  system  quite  difficult 
because  the  developer  is  still  refining  the  operation  of  the  EC  system  to  meet 
the  latest  threat  at  the  expense  of  operational  testing. 

Another  challenge  to  the  OT&E  of  EC  systems  is  the  increasingly  in¬ 
tegrated  nature  of  the  avionics  with  the  host  aircraft.*^  EC  systems  are 
becoming  more  and  more  dependent  on  timely  information  from  other  sensors. 
They  must  be  compatible  with  other  onboard  avionics  and  they  must  com¬ 
municate  with  other  hardware  and  software  functions.  This  means  the  total 


avionics  suite  must  be  up  and  functioning  as  designed  for  a  complete  evalua¬ 
tion  of  the  EC  system’s  contribution  to  the  mission.  This  increase  in  integra¬ 
tion  further  complicates  the  OTA’s  ability  to  test  and  evaluate  the 
effectiveness  and  suitabihty  of  EC  systems.  This  is  due  to  the  requirement  for 
more  representative  test  scenarios  using  both  friendly  and  hostile  players 
along  with  their  associated  command,  control,  and  communications  (C‘M  net¬ 
work  as  well  as  a  multispectral  test  environment.  A  totally  integrated 
avionics  package  also  makes  it  difficult  to  determine  the  contribution  towards 
mission  success  from  the  EC  system  being  tested  or  from  other  onboard 
avionics.  These  challenges  to  the  OT&E  of  EC  systems  must  be  recognized 
and  accounted  for  in  an  effort  to  provide  representative  test  results  to  the 
decision  makers. 

Because  of  these  limitations  and  the  way  EC  test  programs  have  been 
structured,  several  EC  systems  have  failed  to  be  developed  adequately  to  meet 
the  user’s  operational  requirements.  Two  examples  can  be  cited  where  EC 
systems  were  not  adequately  developed  and  tested  before  they  proceeded  into 
the  next  phase  of  the  acquisition  process.  In  the  first  example,  the  Airborne 
Self-Protection  Jammer  (ASPJ)  was  tested  by  the  Air  Force  Operational  Test 
and  Evaluation  Center  (AFOTEC)  and  failed  to  meet  several  operational 
effectiveness  and  suitability  objectives. 

In  this  case,  the  failure  of  the  system  was  due  to  a  combination  of  factors. 
First,  the  system  was  in  development  for  10  years,  during  which  time  the 
threats  it  was  designed  to  counter  were  replaced  by  later  versions.  Conse¬ 
quently,  ASPJ  could  not  counter  the  latest  threats  in  the  operational  environ¬ 
ment.  Second,  immature  or  unproven  technologies  were  used  in  the  design  of 
the  system.  Because  these  technologies  were  not  adequately  tested  early  in 
the  acquisition  process,  problems  with  the  effectiveness  and  suitability  of  the 
system  were  not  discovered  until  it  began  dedicated  lOT&E. 

In  the  second  example,  the  B-IB  defensive  system  was  found  to  be  quite 
limited  in  its  ability  to  counter  the  threats  in  its  projected  mission  scenario. 
Although  this  was  a  presidentially  mandated  program  with  the  decision  to 
produce  the  B-IB  already  made,  the  EC  test  process  failed  to  identify 
problems  with  the  defensive  system  in  the  early  stages  of  the  acquisition 
process. 

Here  again  one  of  the  problems  was  that  new  threats  were  being  deployed 
at  the  same  time  the  defensive  system  was  being  developed.  This  had  a 
significant  effect  on  the  defensive  system’s  ability  to  meet  the  user’s  changing 
requirements.  But  the  greatest  developmental  failure  had  to  be  that  the 
B-lB’s  defensive  system  was  never  fully  tested  in  a  hybrid  ground  test  facility. 
The  first  real  testing  began  in  the  field  .after  the  system  was  instedled  in  the 
B-IB.  Consequently,  the  system  ended  up  in  a  fly-lix-fly  development  cycle 
that  carried  over  to  OT&E.  OT&E  was  not  able  to  break  this  cycle  and 
ultimately  ended  up  supporting  the  developers  and  not  the  acquisition 
process. 

These  two  cases  illustrate  the  lack  of  an  adequate  and  disciplined  test 
process  in  the  early  stages  of  the  acquisition  process.  The  proper  tools  needed 


to  support  these  programs  were  either  not  used  or  not  in  place  when  the 
system  entered  testing.  In  both  cases,  the  OTA  was  relegated  to  a  test-fix-test 
scenario  that  began  daring  DT&E.  As  a  result,  OT&E  did  not  provide  the 
decision  makers  with  what  they  were  looking  for  to  support  program 
decisions. 

Actions  Taken  to  Correct  the  Problems 

Because  of  these  two  EC  systems  and  others  failing  to  meet  the  needs  of  the 
user,  CSAF  Gen  Larry  Welch  initiated  a  “broad  area  review”  of  the  acquisition 
test  process  for  EC  systems.  On  9  March  1988  the  director  of  Strategic/Spe¬ 
cial  Operations  Forces  (SOFVAirlifl  Programs  for  the  secretary  of  the  Air 
Force  (SAF/AQQ),  Maj  Gen  Michael  Hall,  briefed  the  results  of  the  broad  area 
review  which  depicted  the  EC  test  process  as  lacking  in  structure  and  stand¬ 
ardization.^'*  This  led  General  Welch  to  conclude  that  the  acquisition  process 
for  EC  systems  was  broken,  that  EC  systems  took  too  long  to  develop,  cost  too 
much,  and  do  not  always  work.  Further,  Col  Bob  Senko,  chief  of  the  OT&E 
division  at  the  Air  Staff,  reported  that  among  other  problems,  the  CSAF  found 
that  EC  testing  was  at  fault.  Colonel  Senko  directed  the  Air  Force  System.^ 
Command  (AFSC)  commander  to  host  a  group  to  determine  how  to  obtain  a 
logical  test  process  and  define  a  set  of  standard  measures  to  determine  the 
military  worth  of  our  EC  acquisitions.*® 

On  10  October  1988,  after  examining  the  EC  systems  acquisition  test 
process,  the  AFSC-led  ad  hoc  group  released  its  findings  and  recommenda¬ 
tions  in  a  report  titled  “Test  Process  for  Electronic  Combat  Systems  Develop¬ 
ment.”  It  cited  inadequate  testing  as  one  of  the  problems  in  the  acquisition 
process  for  EC  systems.  The  ad  hoc  group  found  that  “an  institutionalized, 
disciplined  process  such  as  that  employed  for  aircraft  and  other  weapon  sys¬ 
tems  does  not  exist  for  the  development  and  upgrades  of  EC  systems.”*®  It 
went  on  to  say  that 

the  lack  of  an  institutional  test  process  manifests  itself  in  our  inability  to  produce 
meaningful  information  for  decision  makers  to  use  in  determining  whether  develop- 
ment/upgradee  should  proceed,  and  in  the  lack  of  discipline  among  programs  as  to 
what  test  reeourceB  should  be  used.” 

In  addition,  a  DOD  Inspector  General  (IG)  report  titled  “Operational  Test 
and  Evaluation  within  the  Department  of  Defense”  states  that 

OT&E  does  not  have  the  major  impact  envisioned  by  the  Congress  on  acquisition 
milestone  decisions  that  was  intended  by  Congress.  Instead  of  using  OT&E  results 
to  delay  or  halt  production  of  weapon  systems  with  questionable  effectiveness  or 
suitability,  acquisition  executives  use  the  results  to  continue  the  test-fix-test 
scenario  begtin  during  developmental  testing.  While  productive  in  supporting  the 
industrial  base  and  ultimately  delivering  new  weapon  By.stems.  the  test-fix-te.  t 
scenario  does  not  meet  congressional  expectations  for  OT&E  to  support  “go-no-go” 


program  decisions.  Thus,  the  OTAE  community  is  often  criticized  by  the  Congress 
and  perceived  as  being  nonsupportive  of  congressional  intent.'* 

The  findings  from  the  ad  hoc  group  and  the  DOD  IG’s  report  highligl^t  the 
need  to  establish  a  disciplined  process  to  acquire  and  test  EC  systems.  Add¬ 
ing  discipline  and  structure  to  the  test  process  will  also  give  the  decision 
makers  timely  information  to  support  production  decisions. 

Decision  makers  believe  that  the  lack  of  a  structured  test  and  evaluation 
process  has  contributed  to  the  problems  and  failures  experienced  in  EC  sys 
tern  development  and  upgrades.  In  a  letter  to  all  MAJCOMs  and  separate 
operating  agencies  ( SOA)  dated  7  November  1988,  the  chief  of  staff  endorsed 
the  ad  hoc  group’s  report  and  directed  immediate  implementation  of  their 
recommendations  for  a  structured  EC  test  process.’®  In  essence,  the  chief  of 
staff  has  tasked  all  Air  Force  organizations  involved  in  developing,  modifying, 
and  testing  EC  systems  to  follow  the  recommendations  outlined  in  the  ad  hoc 
group's  report.  Implementation  of  the  ad  hoc  group’s  recommendations  would 
provide  a  structured  process  characteristic  of  a  scientific  test  approach.  Key 
elements  to  the  scientific  test  approach  include  generating  the  test  require¬ 
ments,  pretest  planning,  carrying  out  the  test,  analyzing  and  evaluating  the 
results,  and  feeding  back  the  test  results  to  validate  and  refine  models  and 
simulations.  This  will  ensure  a  disciplined  approach  to  developing  and  field¬ 
ing  EC  systems.  It  will  also  help  the  OTA  structure  a  test  program  that  will 
avoid  the  test-fix-test  scenario  by  preparing  the  OTAs  for  the  evaluation  and 
by  reporting  the  results  of  testing  to  decision  makers  in  a  timely  manner. 
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Chapter  2 


Test  and  Evaluation  Tools 


Before  outlining  a  structured  electronic  combat  (EC)  test  process,  it  is  im¬ 
portant  to  describe  the  test  and  evaluation  (T&E)  tools  used  to  generate  the 
data  needed  to  perform  an  evaluation  or  assessment.  The  test  manager  must 
be  familiar  with  the  various  types  of  T&E  tools  that  are  available,  their 
purpose,  and  how  they  support  the  EC  test  process.  In  addition,  the  test 
manager  must  be  aware  of  the  assets  and  liabilities  associated  with  each  test 
tool  in  order  to  select  the  proper  mix  of  T&E  tools  for  the  evaluation. 

This  chapter  begins  by  describing  the  most  common  types  of  T&E  tools  used 
to  support  operational  test  and  evaluation  of  EC  systems.  These  tools  can  be 
grouped  into  three  major  categories;  modeling  and  simulation  (M/S),  hybrid 
ground  test  facilities  (GTF),  and  field  test  ranges.  The  advantages  and  disad¬ 
vantages  of  each  tool  are  discussed  as  well  as  their  use  in  supporting  OT&E. 
Of  great  importance  to  the  decision  maker  is  knowing  that  the  test  results 
truly  represent  the  capabilities  of  the  EC  system.  To  provide  this  certainty  in 
the  test  results,  the  T&E  tools  must  go  through  a  process  that  verifies, 
validates,  and  accredits  their  use  in  the  EC  test  process.  This  process  is 
examined  following  the  discussion  on  the  T&E  tools.  The  chapter  concludes 
with  a  comparison  of  the  strengths  and  weaknesses  of  each  tool  and  con¬ 
siderations  in  selecting  the  appropriate  test  tooKs)  for  the  evaluation. 


Modeling  and  Simulation 

The  first  type  of  tool  to  consider  in  the  EC  test  process  is  modeling  and 
simulation.  Models  and  simulations  provide  a  method  to  augment  or  extend 
those  areas  that  cannot  be  evaluated  or  assessed  through  field  testing.  They 
can  overcome  some  of  the  limitations  associated  with  field  testing,  such  as  the 
lack  of  adequate  threat  resources  or  a  field-test  environment  that  is  not  repre¬ 
sentative  of  an  operational  mission,  and  can  easily  accommodate  intelligence 
updates  to  the  threat  definitions.  M/S  can  also  be  used  to  aid  in  the  develop¬ 
ment  of  mission-level  measures  and  evaluation  criteria,  lessen  the  impact  of 
budget  constraints,  and  support  early  involvement  by  the  OTA  in  the  acquisi¬ 
tion  process.  Dr  Patricia  Sanders  is  responsible  for  policy  concerning  the  use 
of  M/S  at  the  Office  of  the  Secretary  of  Defense  (OSD).  She  describes  models 
and  simulations  as 
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tools  that  can  potentially  augment  and/or  complement  actual  field  teats  and  provide 
decision  makers  necessary  information  to  assess  the  progress  of  a  system  toward 
fulfilling  the  operational  needs.  .  .  .  Consequently,  it  is  appropriate  to  use  M/S  as 
an  aid  to  OT&E  planning,  as  an  evaluation  tool  prior  to  the  availability  of  a  com¬ 
plete  system,  and  to  augment,  extend,  or  enhance  actual  field  test  results.' 

Models  and  simulations  will  not  replace  field  testing,  but  they  can  supple¬ 
ment  it  by  providing  the  ability  to  expand  the  operational  evaluation  or  as¬ 
sessment  to  those  areas  where  operational  realism  cannot  be  obtained  in  the 
field.  On  the  other  hand,  using  M/S  to  complement  field  testing  will  be  at  the 
cost  cf  increasing  some  “risk”  to  the  evaluation  because  the  results  from  M/S 
may  not  give  a  complete  picture  of  the  system’s  capabilities.  Even  when 
validated  or  accredited,  M/S  will  generally  yield  higher  risk  and  less  con¬ 
fidence  than  field  testing. 

A  model  represents  a  system  by  simulating  the  operating  logic,  process, 
rules,  techniques,  and  methods  of  a  system  through  a  computer  program. 
Models  aid  in  understanding  or  evaluating  the  operation  and  behavior  of  a 
system.  They  can  give  the  decision  maker  an  idea  of  how  well  the  system  will 
perform  its  job  in  combination  with  other  friendly  systems — ^the  intended  host 
aircraft — and  in  an  operational  environment  made  up  of  threat  systems. 
DOD  5000.2-M,  Defense  Acquisition  Management  Documentation  and  Reports, 
maintains  that 

the  models  used  can  take  a  variety  of  forms,  from  simple  “stubby  pencil  calcula¬ 
tions”  to  elegant  mathematical  formulations  to  large  force-on-force  computer 
simulations.  Clearly,  the  type  of  model  most  useful  for  an  analysis  depends  on  the 
purpose  being  served.’ 

The  process  of  implementing  a  model  is  called  simulation.  Simulations  will 
allow  the  examination  of  a  system’s  behavior  under  selected  conditions.  They 
can  often  demonstrate  the  operating  functions  or  logic  process  when  the  sys¬ 
tem  cannot  be  subjected  to  direct  testing  in  the  field.  As  opposed  to  field 
testing,  simulations  can  provide  a  much  broader  range  of  data  for  evaluation 
at  a  reduced  cost.  For  purposes  of  this  study,  the  terms  modeling  and  simula¬ 
tion  are  often  interchangeable  in  the  context  of  their  use  when  supporting  the 
EC  test  process. 

Models  and  simulations  range  from  those  that  describe  specific  events  to 
those  that  generalize  the  combined  effects  of  many  systems  in  a  large  opera¬ 
tional  scenario.  They  are  organized  by  their  complexity  and  potential  for 
analyzing  a  system’s  contribution  to  combat  operations.  Models  are  generally 
divided  into  four  levels  (table  1).  An  engineering-level  model  examines  the 
technical  performance  of  individual  subsystems  or  techniques  against  distinct 
variables.  Commonly  referred  to  as  a  level  I  model,  its  one-versus-one 
engagements  are  the  most  detailed  of  the  models.  A  level  II  or  platform-level 
model  examines  the  combined  interactions  of  an  aircraft’s  integrated  avionics 
against  several  threat  systems  with  the  application  of  doctrine  and  tactics. 
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The  output  from  a  platform-level  model  represents  an  analysis  of  the  system’s 
effectiveness  when  engaged  with  a  few  threat  s3rstems.  A  level  III  model 
examines  the  mission  effectiveness  emd  suitability  of  a  composite  force  oppos¬ 
ing  many  threats.  Doctrine  and  tactics  for  both  friendly  and  hostile  forces  are 
applied  in  this  mission-level  analysis  to  determine  the  actions  and  reactions 
by  all  the  forces.  The  level  IV  or  campaign-level  model  examines  the  outcome 
of  a  projected  theater  war.  Level  IV  models  incorporate  joint  operations  from 
the  Army,  Air  Force,  and  Navy,  as  required,  in  the  campaign  against  a 
theater-level  force.  They  are  the  most  complex  and  provide  analysis  at  the 
strategic,  policy,  and  force-planning  levels.  In  most  cases  the  results  from 
lower-level  models  can  be  fed  into  the  next  higher-level  model  to  support  its 
analysis  of  the  EC  system’s  effectiveness  and  suitability. 

In  selecting  a  model,  the  test  planner  should  consider  the  complexity  or 
detail  needed  to  address  a  specific  question.  The  model  should  not  be  more 
complex  than  is  absolutely  required.  Decisions  concerning  the  selection  and 
development  of  a  model  shoiild  include  the  user,  developer,  and  tester  early  in 
the  development  of  the  EC  system.  A  plan  shoiild  be  developed  early  on  in  the 
acquisition  process  that  outlines  the  use,  level  of  detail,  and  the  upkeep  for 
the  model.  This  plan  should  show  the  transition  and  growth  of  the  model 
from  the  initial  phase  of  the  acquisition  program  to  its  use  in  supporting  the 
milestone  III  decision  and  beyond.  To  support  the  analysis  of  the  EC  system, 
the  M/S  plan  must  include  a  process  that  shows  how  the  model  will  be  up¬ 
dated,  verified,  and  accredited  for  a  specific  purpose  with  field  test  results.  By 
having  an  M/S  plan  in  place,  the  process  of  planning,  executing,  and  reporting 
the  development  and  use  of  M/S  will  help  provide  credible  results  to  decision 
makers.^ 

To  lend  acceptance  to  the  results,  the  models  and  simulations  must  be 
verified,  validated,  or  accredited  with  help  fiom  the  intelligence  community 
and  field  test  results.  Without  a  verified,  validated,  or  accredited  model  or 
simulation,  the  test  manager  will  have  a  tou^  time  defending  the  use  of  M/S 
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in  addressing  the  critical  operational  issues  (COD.  The  reason  for  this  is  that 
decision  makers  are  looking  for  confidence  that  conclusions  drawn  from  M/S 
are  credible  and  that  valid  conclusions  can  be  formed  about  the  effectiveness 
of  the  actual  system. 

The  advantage  of  a  model  is  that  it  can  be  run  many  times  while  simulating 
a  variety  of  operational  conditions,  thus  accelerating  the  analysis  of  the  EC 
system.  Because  M/S  can  supplement  field  results,  they  help  reduce  the  num¬ 
ber  of  test  missions  or  weapons  expended  in  the  field.  This  can  save  having  to 
pay  for  expensive  field  range  time  or  resources.  Of  course,  there  are  some 
disadvantages  in  the  use  of  M/S.  Models  and  simulations  can  be  expensive  to 
develop,  operate,  maintain,  and  validate  or  accredit.  If  not  validated  or  ac¬ 
credited,  the  decision  maker  may  lack  confidence  in  the  output,  thereby 
degrading  the  tests  usefulness.  But  even  if  the  models  are  vahdated  or  ac¬ 
credited,  they  can  never  provide  the  same  level  of  confidence  in  the  results  as 
could  be  obtained  through  field  tests.  Models  and  simulations  can  require 
long  lead  times  to  develop  and  acquire.  They  can  be  manpower  intensive 
when  in  operation,  and  they  may  not  support  all  the  test  objectives.  The  test 
manager  will  have  to  keep  these  advantages  and  disadvantages  in  mind  when 
determining  the  proper  mix  of  M/S  tools. 

Simulations  are  often  used  during  the  early  development  phases  of  the 
acquisition  process  to  support  the  evolution  of  the  EC  system  and  to 
determine  mission-level  requirements.  During  the  early  design  stages, 
engineering-  and  platform-level  models  can  be  used  to  assist  the  developers  in 
translating  mission-level  requirements  into  system  .specifications.^  A 
system’s  mission-level  requirements  are  identified  throu^  concept  studies  by 
the  MAJCOM  that  has  the  responsibility  for  the  specific  mission  area. 

Because  it  is  not  likely  that  any  representative  hardware  or  software  would 
be  available  during  an  early  operational  assessment,  M/S  can  support  an  EOA 
by  assessing  the  system’3  performance  in  the  planned  operational  scenario.  A 
digital  computer  model  of  the  EC  system  can  be  designed  from  the  initial 
system  design  specifications  and  used  to  assess  its  performance  in  the  in¬ 
tended  operational  environment.  Such  a  model  could  examine  those  critical 
operational  mission  requirements  essential  to  the  EC  system’s  effectiveness 
and  suitability. 

To  support  the  cost  and  operational  effectiveness  analysis  process,  M/S  can 
be  used  in  assisting  the  MAJCOM  in  studying  alternative  concepts  that  sat¬ 
isfy  the  mission  need.  Models  can  estimate  the  performeince  of  a  particular 
system  in  accomplishing  the  mission  objectives.  The  COEA  process  can  also 
benefit  from  M/S  by  using  models  to  help  establish  the  design  and  cost  objec¬ 
tives  before  the  milestone  II  decision.  It  will  be  up  to  the  MAJCOM  respon¬ 
sible  for  the  COEA  to  use  the  same  models  or  updated  versions  in  each  phase 
of  the  acquisition  process  to  perform  an  analysis  of  the  alternatives,  document 
the  results  from  the  cost  anal3r8is  of  various  alternative  solutions,  and  update 
the  life-cycle  costs. 
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As  an  analytical  tool,  AC/S  will  help  ensure  that  the  COIs  are  adequately 
addressed  and  resolved  by  the  end  of  testing.'^  Models  and  simiilations  sup¬ 
port  pretest  planning  in  the  following  areas; 

•  Extrapolating  measures  of  effectiveness  (MOE)  Irom  mission-level  test 
objectives. 

•  Identifying  important  test  parameters  or  elements  to  the  t^t  plan. 

•  Determining  sensitivities  to  various  input  variables. 

•  Chai  arizing  the  operational  test  environment. 

•  Determining  test  resource  requirements. 

•  Assessing  the  impact  of  range  resource  limitations. 

•  Refining  the  test  scenarios. 

•  Identifying  the  required  data  elements  used  to  support  the  evaluation. 

•  Prerunning  the  field  test  scenario. 

•  Structuring  those  flight  test  conditions  that  are  the  most  important  to 
evaluate  in  the  field. 

•  Predicting  potential  outcomes  which  can  be  used  to  verify  test  proce¬ 
dures  and  results  before  the  test  is  conducted  on  the  range. 

During  the  posttest  evaluation,  M/S  can  complement  results  from  field  test¬ 
ing  by  extending  known  parameters  and  actual  test  data  to  other  operational 
scenarios  Also,  by  integrating  actual  test  measurements  into  mission-  or 
campaign-level  models,  M/S  can  extend  test  results  to  address  MOEs  that 
cannot  be  determined  directly  from  field  testing.  If  the  EC  test  process  incor¬ 
porates  the  use  of  M/S  to  augment,  extend,  or  enhance  field  testing,  a 
balanced  blend  of  M/S  and  field  testing  must  be  obtained. 

The  Hybrid  Ground  Test  Facility 

Ideally,  OT&E  would  strictly  use  field  test  ranges  to  evaluate  the  effective¬ 
ness  and  suitability  of  actual  or  production-representative  systems.  Because 
of  the  difficulty  of  subjecting  the  EC  system  to  a  representative  threat  en¬ 
vironment  in  the  field,  a  complete  picture  of  the  system’s  performance 
capabilities  cannot  be  ascertained.  However,  a  hybrid  ground  test  facility  can 
contribute  significantly  to  the  evaluation  of  EC  systems  by  providing  a 
method  to  test  their  performance  in  representative  threat  environments. 

A  hybrid  GTF  can  be  described  as  a  secure  indoor  facility  that  uses 
hardware  and  software  to  simulate  the  tracking,  guidance,  and  communica¬ 
tions  associated  with  threat  systems.  Computer-driven  threat  replicas  or 
simulators  generate  the  representative  radio  frequency  (RF),  infrared  (IR), 
miUimeter  wave  (MMW),  and  laser  emissions  from  threat  systems  for  integra¬ 
tion  into  a  representative  threat  environment.  By  installing  an  actual  EC 
system  in  a  hybrid  GTF  (fig.  2)  and  using  a  computer  to  simulate  the  system’s 
host  aircraft,  testers  can  analyze  a  simulated  operational  mission.  Whether  it 
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Figure  2.  Uninstalled  Ground  Test  Facility 

is  throu^  coaxial  cables,  waveguides,  or  free-space  radiation,  signal  propaga¬ 
tion  between  the  EC  system  and  threat  simulators  can  be  controlled  so  that 
the  emissions  correctly  include  the  effects  of  range,  aircraft  movement,  scan¬ 
ning  antennas,  and  other  factors  that  appear  in  a  real  threat  environment.® 

To  add  a  little  more  realism  to  the  simulation,  human  operators  can  inter¬ 
face  with  the  threat  simulators  through  functionally  correct  controls  and  dis¬ 
plays.  This  will  close  the  action-reaction  loop  between  the  EC  system  and 
threat  simulator.  This  type  of  simulation  is  referred  to  as  closed-loop  simula¬ 
tion  (an  open-loop  simulation  has  no  interaction  with  a  human  operator).  A 
closed-loop  threat  simulator  allows  for  real-time  interaction  between  the 
simulator  operator  and  the  EC  system.  By  installing  EC  systems  in  this 
facility  and  stimulating  them  either  through  digital  connections  or  with  ac¬ 
tual  free-space  radiation,  the  tester  can  then  evaluate  the  EC  system’s 
responses  to  this  stimulation.  This  whole  process  is  referred  to  as  hardware- 
in-the-loop  (HITL)  simulation.  HITL  simulation  allows  the  functional  opera¬ 
tion  of  the  EC  system  under  test  to  interact  with  a  computer-controlled 
simulation  of  a  representative  threat  environment. 

To  conduct  the  test,  the  hybrid  GTFs  require  certain  types  of  data  as  inputs 
to  the  simulation.  This  data  includes  the  fli^t  path  through  the  operational 
environment;  the  host  aircraft’s  radar  cross  section  (RCS);  a  definition  of  the 
EC  S3retem;  antenna  patterns;  and  such  threat  site  data  as  the  location  of  the 
threats,  the  threat  parameters,  the  rules  of  engagement,  and  so  forth.^  In 
addition,  some  hybrid  GTFs  must  have  the  static  and  in-flight  RF  and  IR 
mecusurements  from  the  host  aircraft  as  well  as  the  amtenna  pattern  measure- 
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ments  from  the  onboard  emitters  and  receivers  to  incorporate  into  the  simula¬ 
tion  * 

The  instrumentation  requirements  in  a  hybrid  GTF  are  tailored  to  the 
specific  program  objectives  and  desired  output  products.  Data  is  collected 
through  several  devices  such  as  magnetic  tapes,  paper  tapes,  strip  chart 
recordings,  and  computer  printouts.^  Instrumentation  software  reduces  the 
raw  simulation  data  and  performs  statistical  analysis  on  the  results.  The 
data  products  produced  in  the  hybrid  GTF  include  target  range,  jammer-to- 
signal  ratios,  tracking  errors,  and  missile  miss  distance  information. 

There  are  different  types  and  capabilities  of  hybrid  GTFs  from  which  a  test 
manager  will  have  to  choose.  Each  facility  has  shortfalls  or  limitations  of  one 
kind  or  another.  Hybrid  GTFs  vary  in  capability  with  the  number  and  type  of 
threats  that  can  be  generated,  fidelity  of  the  threat  signals,  and  the  various 
portions  of  the  frequency  spectrum  that  can  be  replicated.  Some  facibties  can 
allow  testing  of  actual  or  production-representative  EC  systems  only  when  the 
system  is  not  installed  in  or  on  their  host  aircraft  (see  fig.  2).  Other  hybrid 
GTFs  provide  a  capability  to  test  with  the  EC  system  installed  in  its  host 
aircraft  (fig.  3).  These  facilities  provide  a  more  representative  indication  of 
how  the  EC  system  functions  while  integrated  with  the  operation  of  the  other 
onboard  avionic  systems. 


GROUND  TEST  FACILITY  ANECHOIC  CHAMBER 


Figure  3.  Installed  Ground  Test  Facility 

The  main  advantage  of  an  uninstalled  hybrid  GTF  is  that  it  gives  the  tester 
an  opportunity  (usually  the  first)  to  evaluate  EC  systems  before  they  are 
installed  on  or  mounted  in  their  host  aircraft.  This  is  useful  when  it  is  just 
the  performance  of  the  EC  system  that  is  being  developed  or  evaluated  and 
there  is  no  concern  for  the  integration  with  the  host  aircraft.  Another  ad¬ 
vantage  of  testing  only  the  EC  system  is  that  tests  can  be  conducted  when  the 
host  aircraft  may  not  be  ready  to  accept  the  EC  system,  or  when  the  interface 
hardware  and  software  between  the  host  aircraft  and  EC  system  may  not  be 
ready  for  integrated  testing. 

The  disadvantage  of  uninstalled  hybrid  GTFs  is  that  they  are  limited  in 
their  ability  to  evaluate  the  integrated  performance  of  the  EC  system  with  the 
host  aircraft’s  avionic  systems.  Without  the  total  avionic  package,  the  tester 
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cannot  assess  any  electromagnetic  interference  (EMI)  or  electromagnetic  com¬ 
patibility  (EMC)  problems  between  the  EC  system  and  the  onboard  avionic 
systems.  Also,  the  uninstalled  hybrid  GTE  does  not  lend  itself  well  to  evaluat¬ 
ing  the  coupling  of  free-space  radiation  to  the  EC  system  sensors  and  assess¬ 
ing  their  performance.  Uninstalled  hybrid  GTFs  cannot  factor  in  genuine 
environmental  effects  associated  with  open-air  test  ranges,  such  as  terrain 
effects,  meteorological  conditions,  or  atmospheric  effects.  Furthermore,  some 
of  the  uninstalled  hybrid  GTFs  have  no  way  to  factor  the  aircrew  member  into 
the  evaluation.  Examples  of  uninstalled  ground  test  facilities  are  the  Air 
Force  electronic  warfare  evaluation  simulator  (AFEWES),  the  real-time 
electromagnetic  digitally  controlled  analyzer  and  processor  (REDCAP),  and 
the  Guided  Weapons  Evaluation  Facility  (GWEF). 

On  the  other  hand,  installed  hybrid  GTFs  provide  the  added  capability  to 
evaluate  EC  systems  installed  on  and  integrated  with  their  host  platforms. 
This  type  of  test  facility  can  include  an  anechoic  chamber  large  enough  to 
accommodate  a  full-scale  aircraft  with  its  EC  system  and  avionic  systems 
installed  (see  fig.  3).  The  anechoic  chamber  would  be  linked  to  the  test  facility 
with  cables  to  provide  mission  control  and  data  collection.  Free-space  radia¬ 
tion  can  then  take  place  in  this  chamber  so  testers  can  evaluate  the  in¬ 
tegrated  avionic  system’s  responses  and  performance  to  this  stimulation.  An 
installed  hybrid  GTF  can  also  consist  of  a  facility  where  antenna  hats  are 
placed  over  the  EC  system’s  sensors  on  a  parked  aircraft.  This  type  of  facility 
does  not  require  an  anechoic  chamber  for  free-space  radiation  of  threat  emis¬ 
sions.  Virtually  any  kind  of  hostile  or  friendly  RF  emission  can  be  generated 
with  the  use  of  signal  generators.  Using  a  computer  to  control  the  generation 
and  radiation  of  RF  emissions,  testers  can  move  transmissions  representing 
the  signals  of  interest  around  the  aircraft  to  make  it  appear  to  the  EC  system 
sensors  that  the  aircraft  is  flying  through  an  operational  scenario.  The  result 
is  that  the  EC  system  responds  to  the  threat  stimulation  as  if  it  were  actually 
fljing  through  a  threat  environment. 

Other  advantages  of  the  installed  hybrid  GTF  are  to  save  time  and  money 
by  identifying  system  performance  problems  and  providing  a  preflight  and 
postflight  checkout  of  the  aircraft’s  avionics.  These  checkouts  allow  the  tester 
to  be  sure  the  EC  system  is  functioning  properly  before  and  after  the  test 
flight.  An  installed  test  facility  can  be  used  to  determine  if  there  are  any  EMI 
or  EMC  problems  between  the  onboard  avionics  that  would  interfere  with  the 
plaimed  test  events.  By  noting  any  EMI  or  EMC  problems,  the  tester  can 
modify  the  scheduled  test  event  to  accomplish  some  other  objective  or  delay 
the  flight  in  order  to  resolve  any  problems. 

There  are  disadvantages  to  installed  hybrid  GTFs.  The  first  is  a  technical 
problem  associated  with  free-space  radiation  in  the  anechoic  chamber.  Due  to 
the  size  of  the  anechoic  chamber,  allowances  must  be  made  for  the  near-  and 
far-field  RF  wave  propagation  effects  and  for  suppressing  any  reflected  en¬ 
ergy.  Furthermore,  because  the  aircraft  is  not  actually  in  free  flight,  target 


dynamics  must  be  factored  into  the  evaluation  and  proper  aperture  polariza¬ 
tion  concessions  must  be  made  as  the  signals  are  moved  around  the  aircraft. 
By  using  antenna  hats,  a  ground  test  facility  can  solve  the  distorted  RF  wave 
patterns,  but  such  devices  do  not  allow  evaluation  of  fuselage-masking  effects. 
Currently,  installed  hybrid  GTFs  cannot  provide  the  multispectral  threat  en¬ 
vironment  needed  to  test  the  performance  of  all  sensor  systems.  Also,  in¬ 
stalled  hybrid  GTFs  do  not  simulate  very  well  the  effects  of  terrain, 
meteorological,  or  atmospheric  conditions.  These  facUities  mainly  employ 
open-loop  threat  simulators,  thus  leaving  the  human  operator  out  of  the  equa¬ 
tion.  Examples  of  installed  test  facUities  include  the  Preflight  Integration  of 
Munitions  and  Electronic  Systems  (PRIMES)  at  Eglin  AFB,  Florida;  the 
Navy’s  Air  Combat  Environment  Teat  and  Evaluation  FacUity  (ACETEF)  at 
Patuxent  River,  Maryland;  and  the  Avionics  Test  and  Integration  Complex 
(ATIC)  at  Edwards  AFB,  California. 

An  aircrew  member  can  be  factored  into  the  simulation  by  linking  the 
aircraft  in  the  anechoic  chamber  or  the  computer-generated  host  aircraft  in  an 
uninstaUed  hybrid  GTF,  to  a  manned  flight  simulator  that  represents  the 
actual  aircraft  cockpit  controls  and  displays.  This  t3q)e  of  simulation  is 
referred  to  as  a  man-in-the-loop  (MITL)  simulation  (fig.  4).  MITL  simulation 
wUl  allow  a  flight  crew  to  fly  a  simulated  mission  through  the  computer¬ 
generated  threat  environment  and  respond  with  appropriate  countermeasures 
and  tactics  as  the  scenario  unfolds.  By  adding  the  aircrew  member  to  the 


MAN-IN-THE-LOOP  SIMULATOR 

Figure  4.  Man-in-the-Loop  Simulation 
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evaluation,  one  can  assess  the  man/machine  interface  and  add  a  little  more 
realism  to  the  operational  evaluation. 

Hybrid  GTFs  are  important  tools  that  can  complement  field  testing  for 
those  areas  that  cannot  be  evaluated  in  the  field.  They  can  supplement  field 
testing  by  providing  complementary  data  and  a  means  to  evaluate  the  effec¬ 
tiveness  of  the  EC  system.  Hybrid  GTFs  can  provide  controlled  and 
repeatable  test  conditions  as  needed  to  satisfy  the  data  collection  require¬ 
ments.  Hybrid  GTFs  can  overcome  some  of  the  limitations  and  shortfalls 
associated  with  field  testing  such  as  cost,  security,  safety,  airspace  and  fre¬ 
quency  restrictions,  a  realistic  operational  threat  environment,  a  limited  num¬ 
ber  of  test  articles,  and  field  range  instrumentation. 

Another  advantage  of  hybrid  GTFs  is  that  they  can  overcome  the  shortage 
of  fielded  threat  simulators  by  providing  a  dense  signal  environment  to 
evaluate  the  EC  system’s  performance.  Hybrid  GTFs  can  be  cost-effective 
because  it  is  impractical  to  replicate  in  the  field  all  aspects  of  an  operational 
environment  due  to  the  cost  to  build,  install,  and  maintain  the  required  num¬ 
ber  of  threat  simulators.  In  addition,  George  Nicholas  states  in  his  article, 
“The  EW  Testing  and  Evaluation  Quandary,”  that  “we  cannot  afford  to  as¬ 
semble  all  the  test  assets  [hostile  and  friendly]  necessary  to  conduct  the  re¬ 
quired  massive  test  exercises  in  the  field.””  Therefore,  hybrid  GTFs  become 
important  tools  when  bridging  the  gap  between  a  representative  operational 
environment  and  the  shortages  of  fielded  threat  simulators.  By  being  able  to 
generate  the  anticipated  signal  densities  of  a  representative  operational  en¬ 
vironment,  the  tester  can  evaluate  the  EC  system’s  ability  to  detect,  identify, 
and  process  threat  signals  and  their  modes  of  operation.  Hybrid  GTFs  can 
provide  a  d)mamic  simulated  combat  environment  that  represents  an  in¬ 
tegrated  air  defense  system.  'They  can  provide  a  method  to  evaluate  the 
effects  of  hostile  and  friendly  interaction,  the  EC  system’s  response  to  the 
threat,  and  the  ssrstem’s  susceptibility  to  electronic  countermeasures  by  both 
hostile  and  friendly  forces. 

EC  systems  are  also  becoming  more  and  more  capable  of  monitoring  and 
exploiting  the  RF,  IR,  MMW,  and  laser  emissions  from  threat  systems,  as  well 
as  processing  huge  amounts  of  sensory  data.  From  this  data  an  accurate 
picture  of  the  operational  environment  can  be  made  by  displaying  the  location 
of  both  hostile  and  friendly  forces  to  the  aircrew.  It  is  this  task  of  processing 
and  assessing  large  amounts  of  sensory  data  in  real  time  that  presents  the 
greatest  challenge  to  the  EC  system.  A  hybrid  GTF  can  significantly  aid  in 
the  evaluation  of  the  EC  system  by  providing  a  realistic  multispectral  threat 
environment  to  test  the  performance  of  the  system  in  fusing  and  correlating 
the  sensory  data,  displaying  it  to  the  aircrew,  and,  if  appropriate,  initiating  a 
countermeasure. 

Hybrid  GTFs  provide  opportunities  to  test  against  secure  features  or  ad¬ 
vanced  threat  systems  where  restrictions  may  exist  in  the  field.  They  can 
provide  a  secure  environment  to  test  highly  classified  systems  and  to  evaluate 
capabilities  against  hostile  data  links  and  the  enemy’s  C^  network. 


As  with  M/S,  hybrid  GTFs  can  support  the  overall  EC  test  process  with 
prefield  testing.  The  test  manager  can  identify  important  test  parameters  or 
elements  and  determine  their  sensitivity  to  various  field  conditions.  Hybrid 
GTFs  can  also  help  in  the  refinement  of  field  instrumentation  requirements 
and  the  data-reduction  and  analysis  techniques,  and  can  assess  the  employ¬ 
ment  of  various  tactics.  In  essence,  the  test  manager  can  use  hybrid  GTFs  to 
assist  in  refining  the  test  plan  to  make  more  efficient  use  of  available  field 
range  time. 

Depending  on  the  EC  test  program,  a  hybrid  GTF  can  provide  the  tester 
with  several  advantages  over  field  testing.  It  will  be  up  to  the  test  manager  to 
clearly  understand  the  test  requirements  before  selecting  the  facility  that  best 
satisfies  the  test  objectives.  Although  results  from  hybrid  GTFs  have  more 
credibility  than  models,  the  threat  simulators  still  need  to  go  through  a 
validation  or  accreditation  process  to  ensure  credible  results. 


Field  Test  Range 


The  final  tool  the  test  manager  needs  to  consider  in  the  EC  test  process  is 
the  field  test  range  (fig.  5).  Here  the  EC  system  is  taken  into  the  field  to 
evaluate  or  assess  its  performance  in  addressing  the  operational  effectiveness 
and  suitability  test  objectives.  Field  testing  provides  the  most  credible  data 
for  OT&E.  It  gives  the  decision  maker  the  opportunity  to  see  how  the  system 
will  function  in  its  intended  environment. 

Field  test  ranges  provide  a  test  environment  that  comes  close  to  a  bat¬ 
tlefield  operational  environment  with  actual  people  and  hardware.  They  sub¬ 
ject  test  articles  to  a  comprehensive,  integrated  array  of  threat  systems  and 
the  doctrine  governing  how  those  s3rstems  would  be  used  by  the  enemy.  Field 
test  ranges  are  made  up  of  reahstic  replicas  of  enemy  equipment  and  a  net¬ 
work  of  C^  systems,  as  well  as  tracking  systems,  instrumentation  and  data- 
collection  systems,  and  the  necessary  command  and  control  network  needed  to 
conduct  a  successful  evaluation.  They  wiU  also  employ  enemy  tactics  and 
doctrine  to  make  the  test  as  representative  as  possible.  Field  test  ranges 
include  the  key  aspects  of  friendly  forces  attacking  or  penetrating  the  enemy's 
integrated  air  defense  network.  They  give  the  tester  an  open-air  environment 
for  evaluating  the  EC  system  that  M/S  and  hybrid  GTFs  cannot  provide. 
However,  to  conduct  an  operational  evaluation  in  the  field  requires  the  use  of 
all  available  threat  simulator  resources,  range  assets,  and  range  instrumenta¬ 
tion  systems  to  generate  the  best  possible,  operationally  representative  test 
scenario. 

Evaluating  EC  systems  in  the  field  environment  provides  the  decision 
maker  with  genuine  results  that  can  come  only  from  an  open-air  environment. 
Field  testing  eliminates  many  of  the  disadvantages  associated  with  a  hybrid 
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Figure  5.  Field  Test  Range 

GTF,  such  as  terrain  effects  or  not  being  able  to  factor  in  the  aircrew 
member’s  decisions.  Field  testing  can  provide  the  data  to  caUbrate  the  digital 
models  and  to  vahdate  and  accredit  the  threat  simulators  in  the  hybrid  GTFs. 
Field  test  ranges  do  not  have  the  technical  problems  associated  with  free- 
space  radiation  as  in  an  anechoic  chamber.  Testing  m  the  field  gives  the 
added  benefit  of  a  complete  end-to-end  evaluation  from  the  sensor  to  the 
aircrew  member’s  display  in  a  dynamic  test  environment. 

Field  test  r^ulges  have  the  advantages  of  examining  certain  effects  that 
cannot  be  adequately  accounted  for  with  digital  M/S  or  hybrid  GTFs.  These 
effects  include 

*  atmospheric  attenuation  and  ducting, 

*  meteorolc^cal  conditions  (wind,  rain,  dust,  etc.), 

*  aircrew  interaction  in  the  test  to  include  time-critical  mission  decisions, 

*  system  interactions  oit  the  host  aircraft  and  other  aircraft  or  resources, 

*  terrain  effects  on  RF  propagation, 

*  interference  with  other  aircraft  in  close  proximity. 
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*  aircraft  structural  masking  of  sensors,  and 

•  actual  antenna  patterns  and  gains  for  each  antenna  or  aperture. 

Field  testing  can  also  have  the  advantage  of  discovering  certain  effects  about 
the  performance  of  the  system  that  may  have  been  dismissed  as  negligible  or 
unimportant  to  the  modelers. 

However,  Lt  Col  Greg  Mann  states  in  his  research  report.  The  Role  of 
Simulation  of  Operational  Test  and  Evaluation,  that  there  are  di.^ndvantages 
to  field  testing. 

It  may  not  be  feasible  to  test  all  aspects  of  the  weapon’s  capabilities  because  of 
range  restrictions,  instrumentation,  safety  constraints,  and  so  on.  It  is  difficult  to 
maintain  the  same  operating  conditions  for  each  replication  or  test.  It  is  time 
consuming  and  costly  to  obtain  the  small  sample  size  and  therefore  have  statistical 
significance.  It  may  be  impossible  to  evaluate  all  or  even  nominal  test  conditions  or 
test  points  in  the  field  tests. 

Other  disadvantages  to  the  field  test  range  are  the  lack  in  numbers  of  repre¬ 
sentative  threat  simulators  and  the  associated  network.  Also,  the  field  test 
range  cannot  keep  up  with  the  deplo3rment  of  the  latest  threat  systems  be¬ 
cause  it  takes  years  to  gather  enough  inteUigence  to  build  a  replica  of  the 
threat  system,  and  it  takes  money.  Another  limitation  to  field  test  ranges  is 
that  the  location  and  laydown  of  the  threat  simulators  are  not  representative 
of  the  operational  environment. 

As  is  the  case  with  ground  test  facilities,  there  are  many  field  test  ranges  to 
choose  from,  depending  on  the  type  and  capabilities  needed  for  making  the 
evaluation.  Examples  of  field  test  ranges  include  the  Electromagnetic  Test 
Environment  (EMTE)  at  Eglin  AFB,  Florida,  and  the  Naval  Air  Warfare 
Center  at  China  Lake,  California. 

Depending  on  the  test,  many  of  the  same  objectives  that  were  evaluated  in 
the  hybrid  GTF  can  be  evaluated  on  the  field  test  range.  However,  to  com¬ 
plete  the  evaluation  of  the  test  objectives  in  vbe  field,  the  test  ranges  have  to 
be  highly  instrumented  in  order  to  generate  the  necessary  data.  Sources  of 
range  data  include  the  manned  threat  simulators  as  well  as  a  time-space- 
position  information  (TSPI)  system  used  to  keep  track  of  the  aircraft  over  the 
range.  This  data  is  recorded  on  magnetic  tape  to  reconstruct  a  time  history  of 
the  test  scenario.  Additional  data-collecting  devices  used  to  record  the  test 
events  and  conditions  during  the  conduct  of  the  test  include  video  tapes,  voice 
recordings,  and  operator  logs. 

Field  tests  produce  data  in  terms  of  target  detection,  missile-firing  oppor¬ 
tunities,  firing  results,  and  system  reliabilities.  They  also  yield  data  that 
includes  weapon  release  signals  from  aircraft,  launch  signals  from  surface-to- 
air  missile  threat  systems,  the  threat’s  mode  of  operation,  and  the  threat’s 
tracking  quality  on  the  target.  By  properly  structuring  the  test,  testers  can 
use  data  from  field  test  ranges  to  verify  previous  ground  testing.  Data  can 
also  be  used  to  validate  and  accredit  the  results  from  the  threat  models  used 
in  the  simulations  and  the  threat  simulators  in  the  hybrid  GTFs,  or  refine 


their  operation.  Furthermore,  correlating  data  from  the  flight  test  ranges 
with  the  models  and  the  threat  simulators  can  improve  confidence  n  the  test 
results  obtained  from  M/S  and  hybrid  GTFs. 

Testing  in  the  field  allows  the  tester  to  address  interoperability  between 
various  weapon  s3rstems  and  to  test  the  efficiency  of  the  man/machine  inter¬ 
face.  Hybrid  GTFs  can  provide  a  good  first  look  at  the  EC  system  in  opera¬ 
tion;  however,  they  cannot  reproduce  the  ’'at  stresses  associated  with  a 
real  operational  environment  for  both  man  a..i^  machine.  Working  under  the 
sti  ess  of  combat  conditions,  the  aircrew  members’  decisions  and  reactions  can 
be  factored  into  the  o  /erall  effectiveness  of  the  EC  system. 

But  the  main  purpose  of  the  field  test  range  is  to  evaluate  the  operational 
effectiveness  and  suitability  of  the  EC  system  in  an  open-air  environment.  By 
conducting  an  operational  evaluation  of  the  EC  system  in  a  field  test  environ¬ 
ment,  testers  can  give  the  decision  makers  an  indication  of  how  effective  it 
will  be  in  supporting  the  user’s  mission  need.  Evaluating  EC  systems  in  the 
field  environment  provides  tremendous  credibility  to  the  evaluation. 

In  addition  to  the  effectiveness  issues,  there  are  the  issues  associated  with 
weapon  system  suitability  that  can  be  adequately  evaluated  only  in  the  field 
environment.  The  reason  for  this  is  that  field  testing  permits  the  evaluation 
of  a  totally  integrated  weapon  system  functioning  as  it  would  in  the  opera¬ 
tional  environment. 

Verification,  Validation,  and  Accreditation 

Models,  simulations,  and  threat  simulators  all  represent  actual  systems 
and  their  operating  environments  within  the  confines  of  a  computer  or  system 
replica.  However,  to  be  an  effective  analysis  tool  in  the  EC  test  pr  /cess,  each 
model,  simulation,  and  threat  simulator  must  go  through  a  verification, 
validation,  and  accreditation  (W&A)  process  to  identify  the  differences  be¬ 
tween  the  model  and  the  actual  system.  The  following  discussion  is  presented 
in  terms  of  a  model,  but  it  cf  apply  to  simulations  and  threat  simulators  as 
well. 

There  are  many  definitions  used  to  describe  verification,  validation,  and 
accreditation.  This  study  offers  the  following  definitions  for  these  terms  as 
proposed  during  the  17  October  1990  Military  Operations  Research  Society’s 
sjrmposium.  Verification  is  “the  process  of  determining  that  a  model  im¬ 
plementation  accurately  represents  the  developer’s  conceptual  description  and 
specification.”’^  Simply  stated,  verification  is  a  process  used  to  ensure  that 
the  model  behaves  as  designed  and  that  the  internal  data,  structure,  and  logic 
that  represent  the  system  are  being  modeled  correctly.’'*  Verification  makes 
no  assertion  about  the  behavior  of  the  model  as  compared  to  the  real  system. 
Validation  is  “the  process  of  determining  the  degree  to  which  a  model  is  an 
accurate  representation  of  the  real  w'orld  from  the  perspective  of  the  intended 


uses  of  the  model.”'®  Validation  is  a  process  that  tests  “the  agreement  be¬ 
tween  the  behavior  of  a  simulation  model  and  the  observed  behavior  of  a  real 
system.”'®  But  validation  is  more  than  just  agreement  between  the  model 
and  the  real  system,  it  must  build  confidence  that  the  model  actually  repre¬ 
sents  the  real  system.  In  his  thesis  “'I'oward  Validation  of  Computer  Simula¬ 
tion  Models  in  Operational  Test  and  Evaluation,”  Capt  James  Arnett  expands 
the  definition  of  validation  in  terms  of  OT&E  to  include  “the  process  of  build¬ 
ing  an  acceptable  level  of  confidence  that  the  simulated  data  agrees  with  the 
real  data  closely  enough  that  an  inference  about  the  simulation  is  a  valid 
inference  about  the  actual  system.”'^  Moreover,  validation  is  an  ongoing 
process  of  building  confidence  in  the  model  to  a  satisfactory  level.  It  requires 
that  over  time  one  can  measure  and  observe  the  same  results  from  the  model 
against  a  known  baseline  of  performance  (e.g.,  threat-detection  range).  And 
finally,  accreditation  is  “the  official  determination  that  a  model  is  acceptable 
for  a  specific  purpose.”'*  Accreditation  requires  that  the  model  must  meet 
specified  measures  of  performance,  provide  confidence  that  the  model  will 
perform  as  designed,  and  be  consistent  over  a  wide  range  of  operation  for  its 
specific  purpose. 

The  test  manager  must  ensure  that  any  anticipated  use  or  development  of 
M/S  for  OT&E  is  documented  in  the  test  and  evaluation  master  plan  (TEMP) 
and  OT&E  test  plan,  with  references  to  a  process  for  verifying,  validating,  or 
accrediting  the  models  used.  Verification  of  models  begins  by  making  sure 
that  the  builder  of  the  model  is  provided  with  the  characteristics  and 
parameters  of  the  threat  as  described  in  the  threat  description  document 
(TDD).  Also  a  clear  statement  of  the  modeling  requirements  is  essential  if 
verification  is  to  occur.  After  verifying  that  the  model  behe  res  the  way  it  was 
designed,  the  test  manager  can  use  data  from  hybrid  GTFs  or  field  testing  to 
validate  the  model.  Hybrid  GTFs  provide  a  good  source  of  data  to  validate  the 
results  from  models,  whereas  field  testing  will  provide  the  best  source  of  data 
to  validate  the  results  from  threat  simulators  in  the  hybrid  GTFs.  The  threat 
simulators  used  in  the  hybrid  GTFs  and  on  the  field  test  ranges  will  be 
validated  by  a  separate  agency  responsible  for  simulator  validation. 

The  process  of  validating  is  much  more  difficult  than  verifying  the  design  of 
the  model.  For  various  reasons  there  will  probably  always  be  differences 
between  the  model  and  the  actual  system.  But  what  is  vital  is  understanding 
the  impacts  of  those  differences  and  making  sure  they  are  documented.  The 
validation  process  requires  comparing  model  results  with  results  from  hybrid 
GTFs  or  field  tests,  so  it  is  critical  to  choose  compatible  scenarios.  Also,  it  is 
important  to  consider  the  type  of  data  that  can  be  used  for  comparison  and 
that  it  is  compatible  with  data  derived  through  hybrid  GTFs  or  field  testing.'* 
Once  validated,  it  is  essential  to  periodically  update  or  provide  feedback  to  the 
model  with  actual  test  data  so  that  it  is  kept  current  and  remains  valid. 

Validation  is  seldom  completed  because  it  is  an  ongoing  process  of  compar¬ 
ing  qualitative  and  quantitative  data  from  field  testing,  hybrid  GTFs,  or 
higher  fidelity  models.  As  a  result,  various  levels  or  parts  of  validated  models 
may  exist.  The  level  of  validation  will  depend  on  the  specific  application. 
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amount  of  data  required,  accuracy,  and  confidence  associated  with  the  valida¬ 
tion  data.^®  It  is  also  dependent  on  the  level  of  detail  required  from  the 
model,  the  vahdation  process,  the  cost,  and  the  required  level  of  agreement 
between  the  model  and  results  from  hybrid  GTFs  or  field  testing.  Since  the 
validation  process  is  ongoing  and  seldom  completed,  accreditation  becomes 
the  level  of  acceptable  validation  needed  for  a  model  when  used  for  a  specific 
purpose.  This  means  that  depending  on  the  decision  being  supported,  the 
level  of  validation  required  for  some  applications  (e.g.,  pretest  planning)  may 
be  at  a  lesser  degree  than  for  other  such  applications  as  performing  the 
evaluation.  Once  the  model  has  been  developed,  fielded,  and  activated,  it 
must  be  maintained,  updated,  and  validated  at  an  acceptable  level  of  per¬ 
formance  throughout  its  life. 

In  addition,  as  the  model  gains  acceptance  as  a  valuable  T&E  tool,  it  is 
often  modified  to  newer  versions  for  use  in  other  test  programs.  If  these 
modifications  to  the  model  are  not  documented  and  controlled,  the  configura¬ 
tion  may  become  unknown  and  the  model’s  value  as  a  T&E  tool  will  be  lost. 
As  a  result,  the  agency  tasked  to  maintain  the  model  must  estabhsh  a  proce¬ 
dure  to  control  and  document  the  configuration.  Controlling  the  configuration 
of  a  model  can  be  just  setting  up  a  procedure  that  ensures  the  model  does  not 
change  without  a  formal  process.  When  the  configuration  does  change,  the 
model  will  have  to  go  through  the  W&A  process  again  to  make  sure  the 
output  is  still  acceptable.  This  can  be  as  simple  as  running  the  model  against 
a  standard  data  set.  The  amount  or  size  of  the  W&A  effort  will  depend  on 
what  was  changed  or  modified  and  by  how  much. 

Proper  Mix  of  Test  Tools 

An  important  part  of  the  electronic  combat  test  process  is  the  selection  of 
the  test  and  evaluation  tools  needed  to  euiswer  the  critical  operational  issues. 
There  is  a  broad  range  of  tools  that  can  be  used  in  the  EC  test  process,  each 
having  its  own  particular  purpose  in  supporting  the  test  program.  It  is  the 
test  manager’s  responsibility  to  select  from  the  box  of  T&E  tools  the  ones  that 
best  support  the  test  program  and  to  ensure  the  tools  are  ready  when  testing 
is  scheduled  to  begin.  These  T&E  tools  must  be  available  and  accredited  at 
the  start  of  each  phase  of  the  EC  test  process. 

There  are  several  possible  ways  of  combining  the  test  tools  to  address  the 
CO  Is.  The  task  is  to  decide  the  best  mix  between  M/S,  hybrid  GTFs,  and  field 
test  ranges.  Some  of  the  test  constraints  or  factors  that  might  be  considered 
in  developing  the  test  approach  and  selecting  the  proper  tools  include  the 
range  of  test  issues,  the  funds  allocated  to  testing,  the  time  allotted  for  test¬ 
ing,  and  the  level  of  confidence  required  in  the  results.^^  Making  a  decision  is 
tough  because  the  test  manager  has  to  commit  to  a  test  approach  before  some 
of  the  details  of  the  test  program  are  known.  He  may  draw  fire  if  one  of  the 
COIs  cannot  be  answered  adequately. 


Test  managers  must  recognize  and  understand  the  relative  strengths  and 
weaknesses  of  each  tool  for  testing.  Capt  William  Farmer  and  Col  John  Nagel 
developed  table  2  with  the  objective  of  rating  several  test  factors  that  affect 
the  capability  of  each  tool  to  effectively  support  OT&E  of  defense-suppression 
systems.^  Although  they  only  addressed  defense-suppression  systems,  the 
ratings  have  the  same  impact  across  the  spectrum  of  EC  system  testing  when 
comparing  the  three  major  categories  of  EC  test  tools. 

By  examining  table  2,  one  can  see  that  a  weak  test  factor  for  one  tool  may 
be  complemented  by  the  strengths  of  another  tool.  For  example,  the  bmited 
number  of  EC  test  systems  for  field  testing  can  be  overcome  with  the  use  of 
M/S  and  hybrid  GTFs.  However,  the  credibility  of  data  will  be  much  better 
with  the  actual  EC  system  used  in  field  testing  than  the  data  derived  from 
M/S.  The  same  argument  can  be  made  for  the  number  and  quality  of  the 
threat  systems.  A  field  test  range  is  a  very  good  place  to  evaluate  the  employ¬ 
ment  of  tactics,  but  M/S  and  hybrid  GTFs  offer  a  better  tool  for  developing 
tactics  and  emplo3rment  concepts.  In  a  hybrid  GTF,  you  have  operator  inter¬ 
action  only  with  the  closed-loop  threat  simulator;  on  a  field  test  range  not  only 
do  you  have  the  interaction  from  the  threat  system  operators  but  the  friendly 
system  operators  can  also  be  factors  in  the  test  scenario.  However,  as  Farmer 
and  Nagel  point  out,  “there  are  sufficient  Lmitations  on  the  range  so  that 
even  in  the  best  test  range  environment  all  of  the  critical  dynamic  interactive 

Table  2 

Test  Tool  Strengths  and  Weaknesses 


Modeling  S 

Hybrid  Ground 

Field  Test 

Test  Factors 

Simulation 

Tost  Facilities 

Rariges 

Identify  Sensitivities 

EC  System 

Very  Good 

Good 

Fair 

Number 

Very  Good 

Good 

Fair 

Creditability 

Fair 

Good 

Very  Good 

Threat  System 

Number 

Very  Good 

Fair 

Fair 

Quality 

Fair 

Good 

Very  Good 

Tactics 

Develop 

Good 

Very  Good 

Fair 

Evaluate 

Fair 

Fair 

Very  Good 

Configuration  Flexfcility 

Good 

Very  Good 

Fair 

Environmental  Realism 

Fair 

Fair 

Good 

Operator  Interaction 

N/A 

Fair  (Throat) 

Good 

(Throat  and  Friendly) 

Systems  Interaction 

N/A 

Fair 

Good 

8ourc«:  Capt  WiWam  0.  Farmar  and  Col  Jofn  F.  Nagel,  ^leckonic  WaHare  Syatem  Operatfonnl  Teat  and  Evdualion,*  final  report 
(Kirtland  AFB.  N.Mex.:  Air  Force  Taat  and  Evaluaion  Cenlar.  March  1980).  21 .  (Reviaed  by  aUhor) 
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processes  which  occur  between  aggressor  and  defender  are  still  not  real- 
ized.”^^  The  important  point  that  should  be  emphasized  from  table  2  is  that  a 
well-planned  EC  test  process  should  use  all  the  test  tools  to  their  full  poten¬ 
tial  in  evaluating  an  EC  system. 

A  further  comparison  of  the  advantages  and  limitations  between  the  three 
categories  of  test  tools  (table  3)  was  presented  in  a  report  on  the  Test  Process 
for  Electronic  Combat  Systems  Development!^*  Table  3  is  broken  out  into  cost, 
capacity,  timeliness,  and  credibility  considerations  for  each  EC  test  tool. 
Capacity  has  to  do  with  the  ability  of  the  test  tool  to  provide  for  force-on-force 
engagements  with  many  hostile  and  friendly  resources  all  taking  part  in  the 
test  scenario.  The  ability  to  easily  change  the  threat  baseline,  as  new  intel¬ 
ligence  on  the  threat  system  becomes  available,  is  also  considered  a  capacity 
of  the  test  tool.  Timeliness  deals  with  time-sensitive  issues  in  the  acquisition 
of  a  system.  Timeliness  is  a  measure  that  determines  how  soon  answers  to 
key  questions  can  be  provided  to  decision  makers.  A  test  tool  must  also  have 
the  ability  to  support  the  evaluation  and  selection  of  one  EC  system  over 
another  when  decision  makers  have  to  make  prompt  decisions  to  improve  the 
survivability  of  a  weapon  system.  Credibility  has  to  do  with  believing  that 
data  from  the  various  test  tools  is  truly  representative  of  the  system  under 
test.  The  data  must  provide  the  necessary  confidence  to  influence  the  judg¬ 
ment  of  key  decision  makers. 

In  table  3,  one  can  see  that  M/S  and  hybrid  GTFs  are  less  expensive  to 
operate.  They  provide  better  capacity  to  evaluate  many  different  force-level 
engagements  in  a  timely  fashion.  However,  the  best  source  of  credible  data 
for  the  evaluation  is  field  test  ranges.  Even  though  the  field  test  range  is 
expensive  and  field  testing  sometimes  occurs  too  late  to  influence  the  acquisi¬ 
tion  decision,  the  data  it  produces  is  still  very  useful  in  the  W&A  process  and 
to  fix  systems  that  do  not  work  ri^t. 

Table  3 

Test  Tool  Advantages  and  Limitations 


TSE  Tools 

Cost 

Capacity 

Timeliness 

Credibility 

Modeling  and  Simulation 

Lowest 

High 

Hours/Days 

Low 

Hybrid  Ground 

Test  Facilities 

Moderate 

Moderate 

Weeks/Months 

Moderate 

Field  Test  Ranges 

Expensive 

Limited 

Application 

Months 

High 

Soure*:  Unilsd  Qteiss  Air  F6rc«  Ad  Hoc  Oroup.  fosf  Pnc^m  hr  B^ehtnic  Comfrof  Sy9t9nw  OevWopmont  vd.  2,  Report  anf 
Apparv§(»a{Mraam  AFa  Wodw^lon.  O.C.:  AFSC/TE.  10  Octabor  10SB).  126.  (RoviMd  by  outwr) 


Ideally,  the  test  manager  will  develop  a  test  approach  that  incorporates  a 
mix  of  T&E  tools  that  generate  as  much  credible  test  data  as  possible  for  the 
decision  maker  at  the  lowest  cost.  To  do  this,  the  test  manager  must  go 
through  a  conceptual  process  of  allocating  funding  against  the  generation  of 
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quality  test  data.^  This  will  help  the  test  manager  focus  on  designing  a  test 
program  that  will  address  the  test  issues  with  the  appropriate  test  tool.  After 
the  test  manager  identifies  the  purpose  and  clearly  spells  out  the  require¬ 
ments  for  each  test  tool,  attention  can  then  be  focused  on  the  cost  of  the 
approach  and  whether  it  answers  the  test  issues  to  the  satisfaction  of  the 
decision  maker.  Finally,  it  is  important  to  involve  the  decision  maker  early  in 
the  test  program  so  that  the  T&E  tools  needed  to  support  his  or  her  decision 
can  be  incorporated  into  the  test  approach. 

By  using  a  disciplined  and  structured  EC  test  process,  the  test  manager 
should  be  able  to  avoid  drawing  fire  by  being  able  to  answer  aU  the  test 
objectives  with  the  best  mix  of  T&E  tools.  Chapter  3  provides  a  detailed 
description  of  the  procedures  for  an  orderly  and  structured  EC  test  process. 
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Chapter  3 


The  Electronic  Combat  Test  Process 


After  becoming  familiar  with  the  teat  and  evaluation  tools  that  are  avail¬ 
able,  the  test  manager  must  next  develop  a  test  procedure  to  verify  that  the 
EC  system  can  satisfy  the  user’s  mission  need.  The  test  procedure  should 
encompass  a  structured  approach  that  incorporates  a  standardized  and  dis¬ 
ciplined  test  process.  It  should  be  designed  in  such  a  way  as  to  provide  an 
estimate  of  the  system’s  effectiveness  and  suitability  in  an  operational  en¬ 
vironment.  Such  a  process  will  then  satisfy  the  decision  maker’s  need  for 
timely  and  meaningful  information  in  each  phase  of  the  acquisition  process 
(see  fig.  1). 

This  chapter  describes  a  structured  EC  test  process  consisting  of  six  dis¬ 
tinct  steps  that  can  be  applied  to  any  EC  test  program.  The  foundation  for 
this  process  is  based  on  a  scientific  test  methodology  with  the  objective  of 
conducting  effective  planning,  test  execution,  and  analysis  of  test  data  to 
measure  the  performance  of  the  EC  system.  This  process,  depicted  in 
figure  6,  can  apply  to  any  stage  of  OT&E. 

Scientific  Test  Methodology 


The  scientific  test  process  allows  for  making  a  tentative  assumption  or  a 
prediction  of  the  system’s  performance  (hjpothesis).  Then,  through  testing 
and  analysis,  a  comparison  is  made  between  the  actual  performance  results 
and  the  predicted  hypothesis.  The  advantage  of  a  scientific  test  process  is 
that  it  provides  structure  and  discipline  to  the  test  while  verifying  or  refuting 
the  hypothesis.  In  their  paper  “AF  EC  Test  Process  for  RF  Receivers,”  George 
F.  McDougal,  Michael  J.  C^per,  and  Dennis  J.  Folds  define  six  steps  in  a 
scientific  test  process.  The  six  steps  are  (1)  derive  the  operational  test  re¬ 
quirements,  (2)  complete  pretest  planning,  (3)  execute  the  test/assessment,  (4) 
process  the  test  data,  (5)  perform  the  posttest  evaluation,  and  (6)  report  the 
results.^  They  also  describe  the  methodology,  the  factors  or  issues  that  relate 
to  the  methodology,  and  the  resulting  products  from  each  step  in  the  T&E  of 
radio  frequency  (RF)  receiver  systems  (fig.  7).  Although  their  paper  centers 
around  a  test  process  for  RF  receivers,  their  fundamental  scientific  test 
methodology  can  be  applied  to  any  operational  assessment  or  evaluation  of  EC 
systems.  The  remainder  of  this  chapter  describes  each  step  in  the  scientific 
test  process  for  EC  systems. 


STEP  1  DERIve  TEST  REQUIREMENTS 

•  MODELING  AND  SIMULATION 
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Figure  7.  The  Scientific  Test  Method 


Deriving  the  Operational  Test  Requirements 


The  first  step  in  the  EC  test  process  is  to  derive  the  operational  test  re¬ 
quirements.  Operational  test  requirements  are  developed  from  the  user’s 
mission  need  or  stated  operational  requirements.  Deriving  operational  test 
requirements  is  extremely  important  because  it  identifies  the  questions  or 
critical  operational  issues  that  need  to  be  answered  and  will  ultimately  result 
in  the  development  of  the  test  objectives.  COIs  are  key  operational  effective¬ 
ness  or  suitabihty  questions  that  must  be  examined  to  determine  if  the  sys¬ 
tem  is  capable  of  performing  its  mission.  Test  objectives  break  down  the  COIs 
into  clearly  defined,  manageable  tasks  or  areas  to  be  examined,  and  they  form 
the  basis  of  an  effective  test  plan.  They  originate  from  operational  require¬ 
ments  that  are  defined  by  the  MAJCOMs  and  stated  in  the  mission  need 
statement  (MNS).  Even  though  there  will  be  cost,  schedule,  and  performance 
trade-offs,  the  test  objectives  will  remain  the  same  at  successive  milestone 
decision  points  and  not  increase  in  number.  After  the  objectives  have  been 
identified,  a  method  must  be  devised  to  evaluate  or  assess  whether  the  system 
can  satisfy  each  objective. 

Deriving  the  test  requirements  begins  by  gathering  together  the  program 
documents  that  describe  the  mission,  the  deficiencies  in  the  current 
capabilities,  and  the  proposed  system  concept.  This  information  can  be  ob¬ 
tained  from  such  sources  as  the  PMD,  MNS,  or  the  operational  requirements 
document  (ORD). 

Once  a  detailed  description  of  the  operational  mission  and  the  proposed 
system  concept  is  provided,  a  mission-level  computer  simulation  of  the 
proposed  operational  scenario  can  be  used  to  further  refine  and  identify  the 
EC  system’s  mission-level  requirements.  Again,  the  MAJCOM  is  responsible 
for  developing  the  mission-level  requirements  and  will  most  likely  develop  the 
mission-level  simulation  for  the  analysis.  Developing  a  computer  simulation 
of  the  operational  scenario  is  not  a  requirement  for  the  OTA.  But  if  available, 
it  can  be  quite  useful  in  determining  the  critical  operational  issues  that  the 
proposed  system  concept  is  designed  to  fill  in  the  stated  mission  need.  With  a 
mission-level  computer  simulation,  analysis  of  postulated  combat  scenarios 
will  lead  to  an  initial  list  of  system  capabilities  required  to  perform  the  mis¬ 
sion.  From  this  list  test  objectives,  along  with  quantitative  and  qualitative 
measures,  can  be  developed.  Eventually,  evaluation  criteria  and  data  require¬ 
ments  for  the  evaluation  can  be  determined.^ 

Measures  are  used  to  answer  the  objective  and  help  to  further  refine  the 
test  planning  process.  They  also  provide  a  link  between  the  test  objective  and 
the  specific  test  method  used  to  evaluate  the  EC  system.  A  measure  provides 
a  quantitative  basis  for  comparing  a  system’s  performance  to  a  specified  re¬ 
quirement.  However,  there  are  two  types  of  measures  that  need  to  be 
developed  to  determine  if  the  system  is  effective  in  meeting  the  mission  need. 
The  first  is  a  mission-level  measure  of  effectiveness  (MOE)  that  determines 
the  overall  degree  of  mission  accomplishment.  MOEs  judge  the  operational 
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effectiveness  and  suitability  of  the  system  under  combat-like  conditions. 
However,  in  most  cases  they  cannot  be  directly  addressed  through  field 
testing. 

The  second  type  of  measure  is  the  measure  of  performance  (MOP),  which 
provides  a  basis  to  determine  how  well  system-level  performance  require¬ 
ments  are  being  met.  MOPs  are  lower-level  measures  that  directly  relate  to 
the  MOEs.  A  change  in  the  sjrstem-level  measure,  such  as  threat-detection 
range,  electronic  countermeasure  technique  assignment,  or  threat  location 
accuracy,  should  have  an  effect  on  the  mission-level  MOE.  Unlike  test  objec¬ 
tives,  measures  will  become  progressively  more  specific  and  increase  in  num¬ 
ber  at  successive  milestone  decision  points.  The  OTA  will  develop  the 
operational  MOEs  and  MOPs  as  the  objectives  for  evaluating  the  EC  system 
become  clear. 

MOEs  are  also  used  in  the  cost  and  operational  effectiveness  analysis 
(COEA)  process  to  show  how  alternative  solutions  compare  in  meeting  mis¬ 
sion  needs  and  to  discriminate  between  the  alternatives.  Generally,  each 
alternative  is  evaluated  against  existing  capabilities  with  MOEs  that  are 
suitable  for  comparison.  It  is  the  MAJCOM’s  responsibility  to  develop  key 
MOEs  to  be  used  in  the  COEA  process.  However,  because  of  the  OTA's  exper¬ 
tise  in  developing  MOEs,  the  OTA  can  support  the  MAJCOM  by  helping 
establish  appropriate  COEIA  MOEs.  COEA  MOEs  translate  measured  or 
predicted  levels  of  performance  for  each  alternative  into  statements  of  relative 
effectiveness.  This  information  is  used  to  verify  that  the  level  of  performance 
assumed  in  the  COEA  has  been  achieved  in  the  actual  system.®  If  the  level  of 
performance  for  the  selected  alternative  is  not  achieved,  then  testers  must 
assess  and  document  how  the  reduced  performance  will  impact  on  the 
system’s  mission  effectiveness. 

DOD  Instruction  (DODD  5000.2,  Defense  Acquisition  Management  Policies 
and  Procedures,  states  that  to  perform  this  assessment,  the  “measures  of 
effectiveness  should  be  developed  to  a  level  of  specificity  such  that  a  system’s 
effectiveness  during  developmental  and  operational  testing  can  be  assessed 
with  the  same  effectiveness  criteria  as  used  in  the  COEA.”^  However,  to  use 
the  same  evaluation  criteria,  there  must  be  a  direct  correlation  between  the 
MOEs  used  to  evaluate  the  operational  effectiveness  and  suitabihty  objectives 
and  those  used  to  determine  the  best  alternative.  Where  there  cannot  be  any 
direct  measurement  to  support  the  MOEs,  MOPs  must  be  derived  that  relate 
system-level  results  to  the  MOEs  and  to  the  COEA. 

Here  again,  a  computer  simulation  that  incorporates  an  EC  digital  system 
model  and  a  definition  of  the  proposed  mission  scenario  can  assist  in  identify¬ 
ing  mission-level  MOEs  and  performance  requirements  in  terms  of  MOPs. 
Modeling  and  simulations  can  also  help  establish  the  quantitative  relation¬ 
ships  between  the  measures  of  effectiveness  and  the  measures  of  perfor¬ 
mance.  In  the  COEA  process,  models  are  used  to  estimate  how  a  particular 
alternative  would  perform  in  accomplishing  the  mission  objectives.  The 
COEA  modeling  effort  is  the  responsibility  of  the  MAJCOM.  If  this  model  is 
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made  available  to  the  operational  test  agency,  then  it  can  be  used  to  help 
determine  the  operational  measures. 

Associated  with  the  test  measures  are  evaluation  criteria.  They  are  the 
standards  used  to  judge  the  achievement  of  operational  effectiveness  and 
suitabihty  requirements.  Evaluation  criteria  consist  of  specified  value  ranges 
for  each  performance  characteristic  in  which  the  measured  values  must  fall  to 
meet  the  MOEs/MOPs.  They  are  tied  directly  to  the  MAJCOM’s  stated  opera¬ 
tional  requirements  and  can  be  derived  through  analysis  of  a  range  of  values 
that  impact  the  performance  of  the  system.  DOD  poficy  requires  that  mean¬ 
ingful  COIs,  test  objectives,  MOEs/MOPs,  and  mission-related  evaluation 
criteria  be  established  before  testing  begins.  This  is  essential  because  criteria 
facilitate  the  development  of  methodology  to  evaluate  and  assess  the  EC 
system’s  ability  to  meet  operational  requirements.  Also,  the  data  require¬ 
ments  and  the  T&E  tools  used  to  generate  the  data  can  be  identified  by 
estabhshing  the  evaluation  criteria.  The  criteria  associated  with  the  COEA 
are  expressed  in  the  form  of  cost  and  performance  thresholds  and  must  be 
clearly  identified  and  explained  prior  to  the  start  of  the  evaluation  or  assess¬ 
ment.  Although  the  OTA  develops  the  test  requirements  and  measures,  the 
responsibility  for  developing  the  evaluation  criteria  for  all  measures  rests 
with  the  user. 

It  is  important  to  point  out  that  using  system  development  contractors  to 
estabhsh  test  requirements  for  operational  testing  beyond  the  low-rate  initial 
production  (LRIP)  decision  is  restricted  by  Title  10,  United  States  Code,  sec¬ 
tion  2399,  “Operational  Test  and  Evaluation  of  Defense  Acquisition 
Programs.”  Title  10  states  that 

a  contractor  that  has  participated  in  (or  is  participating  in)  the  development, 
production,  or  testing  of  a  system  for  a  military  department  or  Defense  Agency  (or 
for  another  contractor  of  the  Department  of  Defense)  may  not  be  involved  (in  any 
way)  in  the  establishment  of  criteria  for  data  collection,  performance  assessment,  or 
evaluation  activities  for  the  operational  test  and  evaluation.^ 

This  means  the  test  manager  must  be  careful  when  using  contractor- 
developed  models  to  support  the  development  of  test  requirements  for  lOT&E 
and  beyond.  The  test  manager  must  monitor  the  development  of  the  models 
to  see  that  they  are  fair  and  that  they  have  been  verified,  vahdated,  and 
accredited.  When  it  is  time  to  run  the  models,  the  test  manager  must  ensure 
that  the  contractor  is  not  involved  with  conducting  or  assisting  in  the  opera¬ 
tion  of  the  model,  or  even  acting  as  an  advisor. 


Completing  the  Pretest  Planning 

As  figure  7  shows,  pretest  planning  entails  a  process  of  planning  a  test  that 
encompasses  the  key  factors  to  the  test  plan  and  making  performance  predic¬ 
tions.  Pretest  planning  consists  of  analyzing  test  constraints,  appl3dng  the 
principles  of  scientific  test  design,  and  specifying  the  test  procedures.®  The 


tester,  by  transforming  the  detailed  test  requirements  into  test  procedures, 
can  develop  a  test  approach  or  plan.  During  pretest  planning  the  testers  will 
make  sure  the  measurable  performance  values  for  each  MOE  or  MOP  are 
clearly  spelled  out  and  that  they  predict  EC  system  performance  for  com¬ 
parison  against  actual  test  results.  When  designing  the  test,  sufficient  test 
points  should  be  planned  to  support  the  comparison  of  test  results  to  the 
pretest  predictions,  the  calibration  of  the  test  tools,  and  the  correlation  be¬ 
tween  the  test  tools. 

Each  of  the  key  factors  to  the  test  plan  are  dependent  on  and  developed 
from  the  derived  test  objectives.  Figure  8  depicts  the  key  factors  that  make 
up  the  test  plan  and  their  relationship  to  one  another.  They  include  opera¬ 
tional  test  scenarios  and  test  conditions,  data  requirements,  test  resource 
requirements,  and  data-processing  and  analysis  requirements. 

The  task  of  setting  up  the  test  scenario  and  selecting  the  test  conditions 
requires  information  on  the  operational  environment  and  the  threats  that 
have  the  most  impact  on  successfully  accomplishing  the  mission.  Part  of  this 
information  is  provided  by  documents  that  state  the  mission  and  the  opera¬ 
tional  scenario.  However,  it  is  the  intelligence  community  that  provides  intel¬ 
ligence  estimates  of  the  representative  threat  environment.  During  pretest 
planning,  the  Air  Force  Intelligence  Command  (AFIC)  and  the  OTA’s  own 
intelligence  division  are  tasked  to  provide  the  technical  data  on  the  threats — 
defining  the  operational  threat  environment  and  validating  the  test 
scenarios.^  They  will  also  supply  thi  eat  information  to  the  appropriate  agen¬ 
cies  for  validating  the  models  and  threat  simulators.  Intelligence  on  the 
threat  environment  will  then  drive  the  development  of  the  operational  test 
scenarios  used  in  the  hybrid  GTFs  as  well  as  the  field  test  ranges.  The  h.sters 
tailor  the  intelligence  support  to  the  specific  test  program  to  determine  the 
optimum  operating  modes  of  the  threat,  its  scan  pattern,  and  the  operating 
frequencies  for  use  in  assembling  the  electromagnetic  test  environment  to 
satisfy  the  test  objectives. 

If  the  test  environment  needs  to  be  realigned  to  match  the  representative 
combat  environment,  M/S  with  the  latest  intelligence  estimates  can  be  used  to 
identify  the  layout  of  the  threat  simulators  for  either  the  hybrid  GTF  or  field 
test  range.  Analysis  of  the  simulation  can  help  optimize  flight  path  options 
for  friendly  and  hostile  aircraft  in  the  scenario.  And  through  computer 
simulation,  testers  can  select  the  sequence  for  the  test  trials  that  best  satisfy 
the  test  objectives. 

A  report  published  by  BDM  International  titled  “ASPJ  Test  Requirements 
and  Test  Concept  Developed  through  Analysis  and  Simulation,”  provides  a 
good  example  of  using  the  computer  simulation,  the  suppression  model,  as  a 
tool  to  develop  an  operational  scenario  and  a  set  of  test  conditions  for  the 
airborne  self-protection  jammer.  This  scenario  included  the  location,  number, 
and  types  of  threat  simulators  on  the  range  and  a  script  identifying  the 
specific  periods  when  the  threat  simulators  would  engage  the  aircraft.  Al- 
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Aerospace  end  Electronics  Con/erence.  vot.  3  (Nei«  YorV:  Punishing  Services.  IEEE).  1342  (revised). 


Figure  8.  Key  Factors  of  Test  Plans 

though  total  reahsm  cannot  be  obtained  on  the  test  range,  the  computer  ^ 

simulation  did  determine  the  most  realistic  operational  test  environment 
possible  given  the  limited  test  resources.  Additionally,  the  simulation  deter¬ 
mined 

•  the  impact  of  EC  support  assets  on  the  mission, 

•  the  conditions  under  which  ASPJ  would  be  required  to  perform,  • 

•  the  specific  threats  that  would  engage  the  aircraft  and  when, 

•  the  engagement  geometries,  and 

•  the  outcome  of  the  engagement.® 
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Once  the  test  scenario  is  identified,  the  next  step  is  to  perform  an  analysis 
of  the  scenario  and  test  conditions  to  derive  the  data  requirements.  Data 
requirements  are  dependent  on  the  MOEs  or  MOPs  and  must  be  identified 
early  in  the  development  of  the  test  plan.  As  depicted  in  figure  8,  the  data 
requirements  will  also  have  some  effect  on  the  selection  of  test  resource  re¬ 
quirements.  The  analyst  tasked  to  perform  the  analysis  is  the  one  who 
develops  the  detailed  test  procedures  that  provide  a  description  and  definition 
of  each  data  element  to  be  collected.  Data  elements  support  the  analysis  of 
specific  performance  requirements  that  are  as-^ociated  with  each  measure. 
Data  is  also  needed  to  monitor  compliance  with  the  test  procedures  and  to 
make  sure  safety  considerations  are  adequate. 

Other  apphcations  for  data  include  quality  control  of  the  data  to  make 
certain  the  test  results  remain  valid,  to  predict  the  performance  of  the  system 
and  refine  the  analysis  tools,  and  to  support  the  correlation  of  test  results 
with  other  test  tools.®  Data  is  also  required  to  document  the  test  environ¬ 
ment,  to  identify  any  anomalies  in  the  expected  results,  to  repeat  the  test 
event  when  duplicating  a  problem,  or  to  analyze  the  test  environment  when 
looking  for  test  assumptions  that  may  be  different  or  that  may  have  been 
violated. 

The  analyst  also  selects  the  required  output  data  products  such  as  print¬ 
outs,  plots,  or  magnetic  tapes  that  will  have  an  effect  on  the  data-processing 
requirements.  The  tendency  to  collect  large  amounts  of  data  must  be 
balanced  with  adequate  plans  for  data  processing  and  analysis.  In  their 
paper  “Data  Acquisition,  Processing,  and  Analysis  for  Electronic  Combat,”  Dr 
David  Culp  and  Arthur  Small  state  that 

when  a  large  number  of  data  variables  are  defined  in  a  requirement,  vast  quantities 
of  data  are  inevitably  collected.  This  causes  a  significantly  increased  work  load  for 
data  acquisition,  data  analysis,  and  data  processing.  The  time  required  to  test, 
analyze,  validate,  and  document  each  additional  data  parameter  is  significantly 
increased. 

A  method  to  overcome  the  collection  of  large  amounts  of  data  is  to  completely 
define  and  specify  the  data  elements.^’  Data  elements  should  be  based  on  the 
test  requirements  and  influenced  by  the  analysis  procedures.  This  will  in 
turn  help  define  the  test  design  and  develop  the  data-coUection  system. 

The  data-collection  system  must  provide  the  analyst  with  enough  data  to 
assess  the  test  results.  Yet  it  is  important  that  the  data-coUection  system  be 
designed  to  minimize  its  impact  on  the  performance  of  the  EC  system.  The 
data-collection  system  must  also  interact  with  the  test  resources  in  a  manner 
that  does  not  interfere  with  the  operational  realism  of  the  test  and  the  quahty 
of  the  data. 

Test  resources  (see  fig.  8)  are  assets  that  support  the  T&E  of  the  EC  sys¬ 
tem.  They  include  the  test  facilities,  models,  threat  simulators,  friendly  sup¬ 
port  assets,  instrumentation  systems,  range  tracking  systems,  mission  control 
centers,  test  facilities,  environment  generators,  and  so  forth.  The  require¬ 
ment  for  a  specific  test  resource  is  driven  by  the  data  requirements  that  are 
used  to  support  the  MOEs  or  MOPs.  Depending  on  which  test  resources  are 
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used  in  the  EC  test  process,  they  can  be  used  to  support  the  development  of 
performance  predictions,  provide  data  for  the  correlation  between  the  T&E 
tools,  and  determine  the  quality  of  the  test  data  that  is  generated. 

The  analyst  is  tasked  to  define  the  test  resource  requirements  needed  to 
support  the  objectives.  Such  detail  as  threat  simulator  and  environment  gen¬ 
erator  requirements,  instrumentation  requirements  for  the  hybrid  GTF  or 
field  test  range,  and  mission-support  assets  (e.g.,  cables,  power  supplies,  cool¬ 
ing  systems,  and  test  equipment  for  hybrid  GTFs;  and  bombs,  missiles,  other 
aircraft,  and  instrumentation  pods  for  field  testing)  must  be  identified.  Stat¬ 
ing  the  resource  requirements  early  in  a  test  program  allows  for  the  iden¬ 
tification  of  shortfalls  that  can  be  incorporated  into  improvement  programs  so 
the  test  resources  will  be  ready  at  the  start  of  the  test. 

There  are  several  different  types  of  instrumentation  systems  that  need  to 
be  considered  during  pretest  planning.  The  first  provides  information  as  a 
function  of  time  during  the  test  mission.  For  example,  on  the  field  test  range, 
an  aircraft’s  position  within  the  test  environment  is  provided  by  time-space- 
position  information  (TSPI).  The  requirement  for  TSPI  is  to  provide  location 
information  on  individual  aircraft  operating  over  a  large  area  at  all  altitudes. 
It  must  also  provide  precision  information  when  determining  threat  system¬ 
tracking  errors  or  missile-miss  distances.*^  In  a  hybrid  GTF,  TSPI  is  not  a 
problem  because  the  aircraft’s  flight  path  is  an  input  value  to  the  simulation. 

Another  example  of  an  instrumentation  resource  that  is  essential  to  the 
test  is  the  spectrum-monitoring  system.  This  system  can  monitor  and  record 
the  operating  frequencies  of  all  ground  and  airborne  emitters  participating  in 
the  test  as  well  as  the  field  test  environment.  Being  able  to  monitor  the 
performance  and  status  of  each  emitter  in  the  scenario  during  the  test  gives 
the  test  director  a  way  to  identify  the  emitters  that  are  not  radiating  or 
operating  as  planned.  Also,  at  the  conclusion  of  the  test,  an  analysis  can  be 
performed  on  the  RF  background  to  assess  any  stray  signals  not  associated 
with  or  interfering  with  the  test. 

The  capability  to  monitor  and  record  switch  actions  is  another  element  of 
the  instrumentation  system.  To  reconstruct  the  test  mission  and  understand 
why  certain  events  occurred,  the  actions  of  each  participant  must  be  recorded. 
This  should  shed  light  on  the  manner  in  which  the  test  resources  were 
operated  and  why  the  test  outcome  occurred  as  it  did. 

The  instrumentation  resources  listed  here  are  not  all  inclusive,  but  are  only 
examples.  The  data  requirements  will  determine  just  exactly  what  kind  of 
instrumentation  resources  are  required  for  the  hybrid  GTFs  or  field  test 
ranges. 

The  final  key  factor  that  must  be  taken  into  account  during  the  develop¬ 
ment  of  the  test  plan  is  the  data-processing  and  analysis  requirements  (see 
fig.  8).  The  data-processing  and  analysis  requirements  are  derived  from  the 
methodology  used  to  answer  the  operational  measures.  The  analyst  is  the  one 
tasked  to  develop  a  data-processing  and  analysis  strategy  that  will  address 
each  performance  characteristic  of  the  system.  Test  conditions,  resources, 
performance  predictions,  the  calibration  of  the  test  tools,  and  the  correlation 
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of  the  results  have  an  effect  on  the  data-processing  and  analysis  require¬ 
ments.  This  section  of  the  plan  should  provide  an  explanation  of  how  the  data 
will  be  collected  and  what  will  be  done  to  the  data  to  transform  it  into  a  form 
for  analysis.  Once  the  data  is  reduced  and  processed,  the  plan  must  show  how 
the  analysis  of  the  test  results  are  conducted  to  address  each  measure.  Usu¬ 
ally  this  amount  of  detail  is  incorporated  into  a  separate  document  called  the 
data  management  and  analysis  plan  (DMAP)  or  placed  in  an  annex  to  the  test 
plan. 

It  is  extremely  important  to  develop  and  test  the  data-processing  and 
analysis  routines  prior  to  the  start  of  testing.  However,  in  practice,  this  is  not 
easy  to  do  because  of  last-minute  changes  to  the  EC  system  or  the  data 
requirements.  This  can  force  new  demands  on  or  changes  to  the  data- 
collection  s}rstem  or  to  the  processing  and  analysis  routines.  To  accommodate 
these  types  of  changes,  the  software  used  to  process  smd  analyze  the  data 
must  be  flexible  enough  to  make  changes  quickly  and  easily,  because  once  the 
test  has  started  much  of  the  time  will  be  taken  in  conducting  the  test,  leaving 
little  time  to  finish  developing  the  data-processing  and  analysis  routines. 

An  important  part  of  pretest  planning  that  is  often  omitted  is  the  genera¬ 
tion  of  performance  predictions.  Performance  predictions  will  help  the  test 
manager  determine  the  test  conditions  that  are  critical  to  the  evaluation  of 
the  s3rstem  and  predict  the  system’s  performance  to  selected  objectives.  These 
predictions  can  assist  in  making  effective  use  of  the  limited  test  resources  by 
allowing  the  test  manager  a  chance  to  survey  the  test  matrix.  This  in  turn 
provides  the  opportunity  to  optimize  the  test  conditions  with  the  limited  test 
resources  and  to  develop  a  set  of  pertinent  operational  test  conditions  for  each 
objective.  Performance  predictions  are  also  used,  to  compare  against 
measured  test  results  when  correlating  the  results  between  the  test  tools  and 
validating  the  models  or  the  hybrid  GTFs  threat  simulators. 

Mission-level  computer  simulations  of  the  operational  environment  can  be 
used  during  pretest  planning  to  help  determine  which  test  conditions  and 
data  requirements  are  needed  to  satisfy  the  test  objectives.  Another  excellent 
application  for  a  computer  simulation  in  pretest  planning  can  be  to  assist  in 

•  designing  the  test  scenario, 

•  setting  up  the  test  environment, 

•  identifying  the  proper  instrumentation  requirements, 

•  determining  the  manning  and  control  of  the  test  resources, 

•  selecting  the  best  sequence  for  the  test  trials,  and 

•  predicting  outcome  values  that  can  be  used  to  compare  with  the  actual 
test  results. 

When  designing  and  setting  up  the  test  scenarios,  a  mission-level  computer 
simulation  can  examine  the  proposed  operational  mission  scenarios.  The 
same  mission-level  computer  simulation  can  be  used  to  identify  the  threats  of 
interest  that  will  make  up  the  test  environment.  In  most  cases,  this  mission- 
level  simulation  will  be  the  same  one  developed  by  the  MAJCOM  and  used  to 
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prepare  the  COEA.  With  this  information  and  the  limited  test  resources  at 
the  hybrid  GTFs  or  field  test  ranges,  a  representative  test  environment  can  be 
developed  that  replicates  to  the  best  of  our  ability  the  combat  environment. 

With  the  help  of  an  EC  digital  system  model,  modeling  and  simulation  can 
be  used  to  predict  the  expected  performance  of  the  actual  EC  system  in  the 
combat  environment.  These  predicted  responses  can  then  be  used  to  compare 
with  the  actual  EC  system's  capabihties  to  detect,  identify,  and  process  threat 
signals  and  mode  changes  in  the  test  environment.  Given  that  the  perfor¬ 
mance  predictions  have  been  accepted  with  reasonable  confidence  when  ac¬ 
credited  and  correlated  with  actual  test  results,  the  test  director  can  then 
extrapolate  the  system’s  performance  to  operations  that  cannot  be  performed 
in  the  field  test  environment.  Also,  the  results  from  the  EC  digital  system 
model  can  be  fed  into  a  system-level  model  of  the  threat  system  to  predict  the 
EC  system’s  effectiveness  in  degrading  the  performance  of  the  threat  system 
and  then  comparing  those  results  to  actual  test  results.  Once  the  test  plan 
has  been  developed  and  approved,  the  next  step  is  to  execute  the  test  or 
assessment. 


Executing  the  Test  or  Assessment 


The  third  step  in  the  scientific  test  process  consists  of  executing  a  sequence 
of  test  trials  and  tasks  designed  to  assure  the  collection  of  quality  data  for  a 
posttest  evaluation  (see  fig.  7).  The  test-execution  tasks  consist  of  instructing 
test  team  and  operator  personnel  on  their  responsibilities,  calibrating  all  ap¬ 
plicable  test  resources  and  instrumentation  systems  prior  to  the  start  of  the 
test  triaUs),  ensuring  the  collection  of  valid  test  data  after  the  test  has 
started,  and  calibrating  the  test  resources  at  the  end  of  the  test.’®  These 
tasks  make  up  a  process  of  controlling  the  quahty  of  the  test  results.  They 
ensure  that  the  test  results  will  not  be  corrupted  by  uncontrollable  factors, 
variations  in  measurements,  and  erroneous  data.  As  the  test  is  being  ex¬ 
ecuted,  the  test  director  is  responsible  for  monitoring  the  test  and  guarantee¬ 
ing  the  collection  of  quality  data.  These  quality  control  tasks  apply  to  both 
the  hybrid  GTFs  and  field  test  ranges,  although  they  may  have  to  be  tailored 
to  the  specific  capabihties  and  resources  in  tiie  hybrid  GTFs  or  the  field  test 
ranges. 

Instructing  the  Test  Team 

Before  the  test  starts,  the  director  must  ensure  that  all  personnel  involved 
understand  the  overall  test  program  and  their  responsibiUties  in  executing 
the  test.  McDougal,  Cooper,  and  Folds  suggest  that  “these  instructions 
should  include  the  scope  of  the  test  plan,  specific  test  objectives,  descriptions 
of  the  test  items,  and  methods  for  conducting  each  test.”’®  These  instructions 
should  also  include  a  description  of  all  signal  parameters  that  the  EC  system 
will  be  sensing  in  the  test  environment,  the  operating  modes  for  the  EC 


system,  the  host  aircraft’s  operational  configuration,  and  the  flight  test 
profile.  In  addition,  the  test  director  must  give  the  facility  or  range  in¬ 
strumentation  system  operators  the  specific  test  conditions  and  sequence  of 
measurements  that  need  to  be  collected. 

Pretest  Calibration 

On  the  day  of  the  test,  the  director  has  to  make  sure  the  test  resources  and 
instrumentation  systems  are  calibrated  and  functioning.  Additionally,  the  EC 
system  under  test  should  be  checked  to  make  sure  that  all  modes  of  operation 
are  functioning  properly  and  that  the  system  will  respond  to  the  test  environ¬ 
ment.  Signal  characteristics  from  each  test  resource  should  be  cahbrated  to 
the  operating  parameters  specified  in  the  test  plan  and  verified  to  be  operat¬ 
ing  properly.  Any  significant  pretest  calibration  results  that  show  a  deviation 
from  what  is  expected  must  be  reported  and  documented  by  the  test  team. 

If  the  calibration  is  off  to  the  point  where  the  test  result  will  be  invalid,  the 
test  director  should  postpone  the  test  until  the  problems  can  be  fixed  or  fly  a 
backup  mission  that  is  not  affected  by  the  problem.’’  Canceling  the  test  is 
also  an  option,  but  range  time  is  too  expensive  and  hard  to  come  by  to  cancel, 
so  the  test  director  should  always  have  a  backup  plan. 

Monitoring  the  Test 

Once  the  test  has  started,  the  director  must  monitor  the  execution  of  the 
test  tc  ensure  it  is  proceeding  as  planned  and  verify  that  sources  of  error  are 
kept  to  a  minimum.  Without  usable  or  reliable  data,  there  is  no  test. 
Methods  the  test  director  can  use  to  confirm  the  collection  of  usable  test  data 
include  near-real-time  monitoring  of  the  test  environment  and  checking  on 
the  quality  of  data  from  the  EC  system.  The  test  director  can  also  compare 
selected  test  results  to  the  predicted  values  generated  during  pretest  plan¬ 
ning.’* 

Near-real-time  monitoring  of  the  test  and  data  is  essential  if  the  test  direc¬ 
tor  is  to  maintain  control  over  the  test.  Near-real-time  monitoring  gives  the 
test  director  the  opportunity  to  ensure  that  the  test  aircraft  are  fl5Tng  along 
their  planned  flight  paths,  that  the  test  resources  are  functioning  properly, 
and  that  instrumentation  systems  are  working  and  producing  quality  data.  If 
it  is  determined  that  the  test  is  not  producing  usable  data  or  not  going  as 
planned,  the  problem  can  be  fixed  or  a  work  around  determined,  or  the  test 
can  be  terminated  without  further  expenditure  t  f  costly  resources. 

Using  quick-look  data  at  the  end  of  each  mission  is  another  way  of  allowing 
the  test  director  the  opportunity  to  assess  the  value  of  the  test  data  and 
progress  prior  to  the  next  scheduled  mission.  This  cursory  look  at  the  data 
reduction  and  analysis  products  may  hig^ight  some  unforeseen  problem  in 
the  data  and  make  it  worthwhile  to  repeat  the  mission  before  the  test  has 
ended.  Performing  a  quick-look  analysis  also  allows  the  test  director  the 
opportunity  to  make  certain  that  the  planned  data  processing  and  analysis 
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routines  will  work  and  that  the  output  products  are  what  is  needed  to 
evaluate  the  EC  system. 

Signal  characteristics  from  the  test  resources  in  the  field  test  environment 
should  be  monitored  and  recorded  during  the  test  to  verify  the  correct  signal 
environment.  It  is  also  important  to  perform  a  spectral  analysis  of  the  test 
environment  to  record  any  unwanted  or  unintended  signal  emissions  that 
mi^t  impact  the  test  results.  Flight  test  profiles  must  be  monitored  and 
recorded  to  ensure  range  safety  and  to  reconstruct  the  mission  profile  for 
posttest  analysis.  These  concerns  are  not  a  problem  when  using  M/S  or 
hybrid  GTFs  because  these  tools  offer  very  controlled  environments.  Finally, 
the  EC  system  must  be  monitored  and  its  output  recorded  for  comparison 
with  the  signal  environment  and  pretest  performance  predictions.'* 

Posttest  Calibration 

At  the  completion  of  the  test,  a  posttest  calibration  of  the  test  resources  and 
the  instrumentation  systems  is  performed  to  ensure  they  are  operating 
properly.  The  calibrated  data  can  then  be  compared  with  pretest  calibration 
results  to  see  if  anything  has  changed  significantly  and  whether  it  would 
impact  the  test  results.  If  there  are  significant  difierences  between  the  pre- 
and  posttest  calibration  values,  the  test  may  have  to  be  reaccomplished. 

Processing  the  Test  Data 

Once  the  “raw”  data  has  been  collected,  the  next  step  is  to  process  it.  All 
data  (e.g.,  TSPI,  emitter  characteristics,  spectrum  analysis,  operator  logs,  and 
data  from  the  EC  system)  that  was  measured  and  collected  must  go  through  a 
data-reduction  process,  where  it  is  assembled  in  a  form  favorable  to  perform¬ 
ing  analysis  and  comparison  to  performance  predictions  or  to  previous  test 
results.  Through  analysis  of  the  data,  the  EC  system’s  level  of  performance 
can  be  quantified  with  respect  to  each  measure.  Initially,  the  analyst  closely 
examines  the  data  to  make  sure  that  it  is  complete  and  that  the  results  are 
consistent  throughout  the  test  trial.  After  the  analyst  determines  that  the 
data  is  complete  and  consistent,  he  or  she  then  formats  and  time-correlates 
the  data  so  that  it  can  be  processed  by  mathematical  operations  in  the  data- 
processing  and  €uialysis  routines.  Since  OT&E  involves  an  event-oriented 
analysis,  the  analyst  must  be  on  the  lookout  for  timing  problems  with  the 
instrumentation  system  such  as  the  accuracy  of  the  timed  events,  or  that  all 
data  sources  are  set  to  the  same  reference  time. 

With  the  help  of  computer  processing  to  run  both  the  data-processing  and 
analysis  routines,  processing  of  large  amounts  of  test  data  can  be  sped  up. 
But  even  though  computers  can  speed  up  the  processing  of  the  data,  they  can 
stiU  take  days,  weeks,  or  months  to  complete  the  job.  The  time  to  process  the 
data  win  depend  on  the  following  factors; 


*  Is  the  data-reduction  equipment  at  the  same  location  where  the  test  is 
being  conducted? 

*  Does  the  raw  data  have  to  be  shipped  to  some  other  location  for  pro¬ 
cessing? 

*  Are  the  data-reduction  and  processing  routines  complete  and  functioning 
properly? 

*  Is  the  data  scattered  over  a  wide  geographic  area? 

*  Are  there  enough  personnel  to  process  and  analyze  the  results  in  a 
timely  manner? 

Each  factor  will  have  some  impact  on  the  time  it  takes  to  get  the  results  and 
will  have  to  be  considered  when  determining  the  time  needed  to  process  the 
test  data. 

Depending  on  which  stage  of  OT&E  the  program  is  in  and  the  T&E  tool 
being  used,  the  processed  data  can  assist  in  selecting  performance  criteria  for 
further  testing,  facilitate  the  planning  of  any  subsequent  test,  or  correlate  test 
results  between  various  test  tools.^®  Furthermore,  the  results  from  the 
processing  step  can  be  used  as  a  source  of  quick-look  data  to  assure  the 
collection  of  quality  data.  As  soon  as  the  test  data  is  processed,  a  posttest 
evaluation  of  the  EC  system’s  performance  can  begin. 

Performing  the  Posttest  Evaluation 

The  fifth  step  in  the  scientific  test  process  is  to  perform  the  posttest  evalua¬ 
tion.  It  is  composed  of  anal3rzmg  the  EC  system’s  performance,  comparing 
test  results  to  pretest  predictions,  extrapolating  the  test  results  to  other 
operational  scenarios,  correlating  the  results  between  the  test  tools,  and  sum¬ 
marizing  the  processed  data  collected  from  the  test  trial(s)  (see  fig.  1)}^  Test 
results  representing  system-level  performance  are  compared  with  the  evalua¬ 
tion  criteria  to  determine  if  the  system  met  or  did  not  meet  the  MOP.  In 
addition,  the  test  results  are  compared  against  predicted  performance  results 
generated  during  pretest  planning.  'The  reason  for  comparing  the  test  results 
with  the  model  predictions  is  for  validating  or  accrediting  the  model  for  use  in 
estimating  mission-level  outcomes  or  measures. 

If  there  are  any  differences  between  the  predicted  and  measured  values,  a 
determination  must  be  made  as  to  whether  there  is  a  lack  of  fidelity  in  the 
test  resotirces  or  computer  simulations,  defects  in  the  test  design,  or  problems 
in  the  performance  of  the  EC  system.^^  When  the  test  resources  or  computer 
simulations  lack  the  needed  fidelity,  they  must  be  updated  to  reflect  the 
correct  performance  of  the  threat  system,  based  on  current  intelligence  es¬ 
timates,  and  the  test  trials  repeated  as  necessary.  When  the  source  of  the 
differences  is  the  test  design,  the  test  may  have  to  be  restructured  and  rerun. 

Throu^out  the  EC  test  process,  the  extrapolation  of  performance  predic¬ 
tions  to  operational  scenarios  is  ultimately  contingent  upon  comparing, 
validating,  and  accrediting  the  pretest  predictions  with  field  testing  (fig. 


9).  Validation  and  accreditation  of  performance  predictions  are  essential 
when  extrapolating  test  results  to  operational  scenarios  that  cannot  be 
evaluated  in  the  test  environment.  Predicted  and  measured  values  should 
match  well  enough  to  provide  high  confidence  in  the  EC  system  model. 
During  the  posttest  evaluation,  any  W&A  that  takes  place  must  be  docu¬ 
mented  in  order  to  confirm  the  degree  of  validity  of  the  model  results. 

Even  when  range  results  match  fairly  well  with  the  pretest  predictions, 
comparing  test  results  obtained  from  the  different  test  tools  and  supplying  the 
proper  feedback  to  calibrate  the  test  tools  provide  credibihty  to  the  EC  test 
process.  A  disciplined  test  process  will  compare  the  test  results  obtained  in 
the  hybrid  GTF  with  those  obtained  from  computer  simulations.  By  the  same 
token,  field  test  results  should  be  compared  with  the  test  results  obtained 
from  both  the  hybrid  GTF  and  computer  simulations  (see  fig.  9).  Calibrating 
the  performance  predictions  with  the  measured  test  results  during  each  stage 
of  the  test  process  will  improve  the  accuracy  of  future  performance 
predictions. 

It  is  during  the  posttest  evaluation  that  the  correlation  is  made  of  the  test 
results  obtained  from  using  each  of  the  test  tools  under  similar  test  condi¬ 
tions.  The  degree  of  correlation  obtained  between  the  test  tools  will  influence 
the  validity  of  the  test  results.  Once  the  test  results  have  been  correlated 
with  the  T&E  tools,  and  the  pretest  performance  predictions  have  been 
validated  and  accredited,  the  test  director  can  then  use  the  M/S  tools  to 
generate  updated  performance  predictions  for  any  subsequent  testing.  The 
test  director  can  also  answer  the  mission-level  MOEs  by  using  the  models  to 
extrapolate  the  EC  system’s  performance  to  the  operational  scenario. 

In  most  cases,  the  system-level  measures  can  be  answered  through  direct 
application  of  the  processed  field  test  results.  However,  the  primary  purpose 
of  the  OT&E  is  to  measure  the  system’s  contribution  to  mission  accomplish¬ 
ment  rather  than  just  determining  system  performance.  Because  of  the  re¬ 
quirement  to  estimate  the  EC  system’s  contribution  to  the  mission,  there  will 
be  cases  where  mission-level  MOEs  cannot  be  determined  through  direct  ap¬ 
plication  of  field  test  results.  This  is  due  to  the  fact  that  certain  test  condi¬ 
tions  cannot  be  replicated  in  the  field  test  environment.  Limitations  such  as 
shortages  of  test  resources,  a  less-than-representative  test  environment,  or  an 
EC  system  that  is  still  evolving  will  have  a  bearing  on  reproducing  a  realistic 
operational  test  scenario  to  answer  mission-level  MOEs.  'Therefore,  to  be  able 
to  answer  the  mission-level  MOEs,  the  analyst  may  have  to  return  to  the 
same  computer  simulation  that  was  used  in  designing  the  test  to  determine 
the  EC  system’s  effectiveness  in  its  intended  operational  environment.  Feed¬ 
back  of  field  test  results  to  calibrate  and  accredit  the  models  is  essential  for 
this  effort  to  succeed. 

The  analyst,  by  producing  field  testing  results  that  validate  and  accredit 
the  performance  predictions  at  a  system  level  of  detail,  can  extrapolate  the  EC 
system’s  performance  in  a  representative  operational  scenario  using  mission- 
level  simulations.  Of  course  when  using  models  to  extrapolate  performance  of 
the  system,  it  will  be  at  the  cost  of  increasing  “risk”  to  the  evaluation.  Risk 
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Figure  9.  Extrapolating  Test  Results  to  Operational  Scenarios 

comes  from  the  fact  that  there  will  always  be  some  uncertainty  that  the 
models  wrill  give  a  true  picture  of  the  system’s  capabilities.  Hiis  is  going  to 
always  be  true  even  if  the  models  or  simulations  have  gone  through  the 
W&A  process. 

It  is  also  during  posttest  analysis  that  the  MAJCOM  will  review  and  up¬ 
date  the  COEA  to  determine  whether  the  selected  alternative  is  still  the  most 
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cost-effective  approach  in  satisfying  the  operational  requirement.  This  is 
done  by  comparing  measured  test  results  to  those  predicted  for  the  COEIA 
measures  of  effectiveness  and  performing  a  sensitivity  analysis  to  determine 
whether  the  EC  system  is  still  the  optimum  alternative.  The  OTA  can  assist 
the  MAJCOM  in  the  COEA  process  by  proAnding  operational  test  results  for 
the  MAJCOM  analj^is. 

Posttest  evaluation  for  each  acquisition  phase  is  complete  when  the  values 
for  each  measure  are  determined,  the  differences  between  the  predicted 
results  and  actual  results  are  resolved,  and  the  test  results  are  documented 
and  recorded  along  with  the  processed  test  data.  The  test  results  can  support 
other  studies,  EC  test  programs,  and  the  decision  makers  when  they  have 
been  documented  and  recorded.  If  further  testing  is  required,  results  from  the 
posttest  evaluation  should  be  fed  back  to  update  the  test  tools  and  to  reac¬ 
complish  pretest  planning  for  current  or  follow-on  phases  of  the  EC  test 
program. 

Reporting  the  Results 

The  final  step  in  the  scientific  test  process  is  to  report  the  results  to  key 
decision  makers  in  the  acquisition  process,  to  their  staffs,  and  to  the 
MAJCOM  so  they  can  be  acted  upon.  The  report  should  be  organized  in  such 
a  way  as  to  get  needed  information  into  the  proper  hands  on  time.  The  final 
report  must  be  organized  and  based  on  what  type  of  information  is  needed, 
when,  and  for  what  purpose.  The  final  report  should  provide  an  executive 
summary  up  front  that  summarizes  the  results  in  enough  detail  to  support  an 
acquisition  decision.  The  remainder  of  the  final  report  provides  a  permanent 
record  and  audit  trail  of  significant  OT&E  data,  limitations,  results,  con¬ 
clusions,  and  recommendations.  The  final  report  should  be  written  objectively 
and  relate  the  test  results  to  user  requirements.  In  a  1986  report  to  the 
Senate  Committee  on  Governmental  Affairs  entitled  Weapon  Performance: 
Operational  Test  and  Evaluation  Can  Contribute  More  to  Decisionmaking,  the 
General  Accounting  Office  recommends  that  the  final  report 

1)  state  whether  OT&E  demonstrated  that  the  system  met  operational  require¬ 
ments,  2)  discuss  the  operational  effect  of  significant  test  limitations  and  adverse 
test  results  on  system  performance,  and  3)  clearly  state  whether  the  system  tested 
is  operationally  effective  and  suitable.^ 

The  final  OT&E  report  must  be  approved  and  signed  no  later  than  60  days 
after  the  last  test  event  as  required  by  AFR  55-43,  Management  of  Opera¬ 
tional  Test  and  Evaluation?*  If  the  final  report  is  not  ready  for  release  45 
days  before  the  scheduled  milestone  decision,  an  interim  summary  report  is 
authorized.^  However,  unless  the  posttest  evaluation  is  complete,  an  interim 
summary  report  should  not  be  used  to  support  an  acquisition  decision.  In 
most  cases,  an  mterim  report  is  based  on  interim  results  instead  of  a  complete 
anal3nsis  of  the  test  results  to  include  the  necessary  feedback  to  the  models. 


Therefore,  it  should  be  used  only  in  those  cases  when  the  analysis  is  complete 
and  the  final  report  cannot  be  published  in  time  for  distribution.  The  interim 
summary  report  is  usuaUy  in  the  form  of  a  message  that  summarizes  the 
OT&E  findings  in  enou^  detail  to  support  the  decision  makers.  The  techni¬ 
cal  information  generated  during  the  test  and  used  to  support  the  conclusions 
should  be  documented  in  separate  data  documents. 

A  formal  briefing  may  be  used  in  support  of,  or  in  some  cases,  instead  of, 
the  final  report.  The  briefing  may  be  presented  as  an  executive-level  presen¬ 
tation  to  key  decision  makers  and  other  interested  agencies  (i.e.,  operating, 
supporting,  and  participating  commands). 

This  concludes  the  description  of  a  structured  scientific  test  process  for  the 
evaluation  of  EC  S3rstem8.  It  is  a  process  that  can  be  tailored  to  the  specific 
EC  test  program  and  applied  to  any  stage  of  OT&E.  Depending  on  the  status 
of  the  test  program  and  the  acquisition  phase,  portions  of  the  scientific  test 
process  may  not  need  to  be  reaccomplished.  For  example,  deriving  MOEs  may 
not  be  necessary  as  the  program  enters  into  the  engineering  and  manufactur¬ 
ing  development  phase  because  the  MOEs  should  have  been  determined 
during  the  previous  phase.  Of  course  there  are  always  exceptions,  and  a 
review  of  what  has  been  accomplished  as  well  as  an  understanding  of  the 
objective  for  the  particular  stage  of  OT&E  will  give  the  test  manager  an 
indication  of  the  appropriate  steps  that  need  to  be  accomplished  in  the  EC 
test  process. 

It  must  be  understood  that  by  following  a  disciplined  test  process,  OTAs 
must  plan  for  sufficient  time  and  money  at  the  conclusion  of  the  test  to  feed 
the  data  back  into  the  models  and  hybrid  GTFs.  It  also  means  that  the  OTA 
cannot  use  the  money  programmed  for  providing  feedback  to  the  models  and 
extrapolating  test  results  for  extra  flight  tests  when  there  is  a  problem  to 
resolve  and  a  milestone  decision  point  coming  up.  A  disciplined  test  process 
also  provides  the  shortest  method  to  acquire  quality  systems.  But  if  shortcuts 
are  taken  or  part  of  the  process  is  left  out  to  save  time  or  money,  the  test 
process  invariably  will  take  more  time.  By  sticking  with  a  disciplined  test 
process,  the  OTAs  will  be  able  to  standardize  the  OT&E  of  EC  systems. 

Now  that  a  basic  description  of  the  T&E  tools  and  the  steps  to  the  scientific 
test  process  have  been  presented,  it  is  time  to  see  where  the  tools  are  used  to 
support  each  phase  of  the  acquisition  process.  Chapter  4  describes  the  tasks 
to  be  accomplished  by  the  system  program  office  and  operational  test  agencies 
when  appljring  the  T&E  tools  in  a  scientific  test  process  to  support  EC  system 
program  decisions. 
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Chapter  4 


Test  and  Evaluation  in  the 
Acquisition  Process 


Each  phase  of  the  acquisition  process  provides  a  way  to  advance  new  find¬ 
ings,  ideas,  or  opportunities  in  transforming  stated  mission  needs  into  well- 
defined,  system-specific  requirements.  Milestones  are  established  at  the  end 
of  each  phase  to  assess  the  program’s  status,  to  assess  the  acquisition  plan  for 
the  next  phase,  to  manage  risk,  and  to  decide  if  the  program  should  be  con¬ 
tinued,  redirected,  or  terminated.^ 

This  chapter  defines  the  tasks  to  be  accomplished  within  each  phase  of  the 
acquisition  process,  then  describes  the  focus  of  the  system  program  office 
responsible  for  developing  the  electronic  combat  system.  It  also  shows  where 
operational  test  and  evaluation  is  conducted  to  support  the  acquisition  process 
and  the  type  of  information  provided  to  the  decision  makers  at  each  milestone 
decision  point.  Finally,  this  chapter  points  out  where  the  T&E  tools  are  used 
to  support  the  EC  test  process  within  each  phase.  We  begin  by  determining 
the  mission  need,  then  proceed  through  each  phase  in  the  acquisition  process 
(fig.  10). 


Determining  Mission  Need 

Before  entering  the  concept  exploration  and  definition  phase,  the 
MAJCOMs  will  have  to  document  how  the  current  capabilities  are  deficient  in 
meeting  the  mission  need.  If  a  solution  cannot  be  reached  throu^  non¬ 
material  options  such  as  changing  operational  doctrin ',  tactics,  or  training, 
then  a  mission  need  statement  describing  the  deficiency  in  broad  terms  is 
prepared.  It  will  be  the  Joint  Requirements  Oversight  Council’s  job  to  review 
the  MNS  and  confirm  that  a  nonmaterial  solution  is  not  available.  The  coun¬ 
cil  has  the  authority  to  validate  the  MNS  and  forward  it  to  the  milestone 
decision  authorities  for  their  use  in  making  program  decisions.  Once  the 
mission  need  has  been  validated,  the  objectives  of  the  EC  system  can  be 
established  as  well  as  the  minimum  acceptable  performance  requirements. 
These  requirements  will  form  the  basis  of  the  requirements  process  and 
remain  consistent  with  the  initial  MNS.  The  quantitative  and  qualitative 
performance  requirements  that  are  established  for  the  system  vnll  be  docu¬ 
mented  by  the  MAJCOM  in  the  operational  requirements  dociiment  (ORD). 


The  ORD  will  also  contain  a  description  of  how  the  S3rstem  will  be  operated, 
deployed,  and  supported  by  the  MAJCOM. 

As  the  program  progresses  through  successive  milestone  points,  the 
measures  and  s}r8tem  performance  requirements  will  increase  in  number  and 
become  more  detailed.  It  is  here  that  M/S  can  be  used  to  assist  in  the  require- 
ments  process  by  deriving  the  initial  mission-level  requirements  and  analyz¬ 
ing  the  cost  and  performance  trade-offs.  Furthermore,  as  the  EC  system 
progresses  through  each  phase  of  the  acquisition  process,  M/S  can  be  used  to 
update  the  requirements  prior  to  each  milestone.  The  MAJCOM  will  docu¬ 
ment  any  updates  or  changes  to  the  requirements  in  the  ORD. 

In  addition  to  identifying  mission  needs  or  deficiencies,  the  MAJCOM  will 
be  summariziog  the  threats  and  the  projected  threat  environment  in  the 
MNS.  The  basis  for  this  information  rests  with  Defense  Intelligence  Agency 
(DIA)-produced  and  -validated  threat  projections  that  extend  10  to  20  years 
into  the  future.  Furthermore,  the  DIA-produced  threat  documents  will  be 
used  to  support  a  system  threat  assessment  and  to  determine  the  program 
costs  and  operational  effectiveness  estimates. 

Concept  Exploration  and  Definition 

The  main  objective  of  the  concept  exploration  and  definition  phase  is  to 
study  all  proposed  solutions,  determine  key  system  performance  and  design 
characteristics,  and  appraise  the  operational  capability  and  effectiveness  of 
each  solution.  It  is  here  that  design  trade-off  studies  take  place  to  analyze 
proposed  solutions  that  satisfy  the  need  stated  in  the  MNS.  Computer 
simulations,  past  experience,  and  knowledge  from  previous  testing  assist  in 
defining  and  selecting  a  preferred  system  concept.  The  proposed  technology 
to  meet  the  operational  need  is  assessed  to  see  if  it  is  achievable  and  to 
determine  key  mission-level  performance  requirements.  It  is  during  this 
phase  that  an  electronic  combat  digital  system  model  may  be  developed  by  the 
system  program  office  to  study  proposed  alternatives.^  This  model  will  have 
enou^  detail  to  be  used  in  a  mission-level  simulation  to  make  design  studies 
and  to  help  identify  the  most  promising  concept.  One  of  the  keys  to  the  EC 
test  process  is  that  this  model  can  be  used  to  support  the  requirements 
process.  Once  developed,  the  EC  digital  syrstem  model  should  be  usable  to 
determine  the  mission-level  requirements  throughout  the  life  of  the  system. 

During  this  phase,  the  SPO  may  also  use  mission-level  computer  simula¬ 
tions  developed  by  the  MAJCOM  or  other  agencies  to  analyze  the  operational 
scenario  and  the  predicted  performance  of  proposed  alternatives.  In  addition, 
breadboard  components  of  the  proposed  technologies  or  prototype  hardware 
can  be  installed  in  airborne  laboratories  to  generate  data  in  an  open-air  en¬ 
vironment.^  It  is  important  to  keep  in  mind  that  the  SPO  will  not  be  testing 
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or  evaluating  any  actual  EC  system  hardware  from  the  proposed  solutions. 
EC  system  hardware  will  not  be  examined  until  tlie  next  phase  of  the  acquisi¬ 
tion  process. 

Testing  technology  protot3rpes  in  an  open-air  environment  provides  the  op¬ 
portunity  to  demonstrate  the  performance  of  the  proposed  solutions  in  a  more 
realistic  test  environment  than  can  be  obtained  in  a  ground-based  laboratory. 
The  data  generated  from  the  airborne  laboratory  can  be  used  to  assess  the 
performance  of  candidate  technologies,  update  the  system  models,  and  com¬ 
pare  the  performance  with  the  pretest  predictions  from  the  computer  simula¬ 
tions. 

Along  with  inputs  from  the  OTA,  the  SPO  will  identify  the  initial  test 
resources  that  are  required  to  support  developmental  and  operational  testing. 
Any  deficiencies  will  be  pointed  out  to  the  test  investment-planning  process. 
Identifying  the  deficiencies  tliis  early  in  the  program  should  give  enough  lead 
time  to  make  available  the  necessary  test  resources  for  developmental  and 
operational  testing. 

During  the  concept  exploration  and  definition  phase,  the  OTA  will  be 
fashioning  an  early  operational  assessment  for  the  specific  EC  system.  The 
EOA  can  vary  from  program  to  program  depending  on  the  objective  and  level 
of  involvement  of  the  OTA.  Throughout  this  phase,  the  OTA  will  be  respon¬ 
sible  for  assessing  the  operational  impact  of  the  proposed  solutions,  providing 
an  advisory  input  to  the  source-selection  officials,  and  developing  an  overall 
OT&E  strategy  for  the  system.^  This  is  also  the  point  where  critical  opera¬ 
tional  effectiveness  and  suitability  issues  are  developed  to  become  the  focus 
for  OT&E.  These  COIs  will  be  generated  from  the  user’s  stated  requirements 
and  will  provide  the  basis  for  the  test  objectives.  Furthermore,  because  of  the 
possibility  of  using  contractor-  or  MAJCOM-developed  models  or  simulations, 
the  OTAs  would  be  remiss  if  they  did  not  monitor  the  M/S  development.  By 
monitoring  the  development  of  the  models,  the  OTAs  can  ensure  that  the 
models  are  impartial  and  accredited  for  their  specific  use. 

To  support  the  milestone  I  decision,  the  MAJCOM  develops  a  COEIA  that 
includes  a  broad  range  of  alternative  solutions  that  satisfy  the  mission  need. 
AFR  57-1,  Air  Force  Mission  Needs  and  Operational  Requirements  Process, 
states  that  “the  COEA  should  define  the  performance  and  operational  charac¬ 
teristics  most  affecting  mission  accomplishment  so  program  design  and  cost 
objectives  can  be  established  for  Phase  I.”®  The  COEA  should  also  clearly 
spell  out  why  acquiring  a  new  system  is  preferred  over  modifying  an  existing 
system.  These  early  estimates  are  expected  to  be  quite  rough  due  to  the 
difficulty  of  obtaining  accurate  organizational  and  operational  cost  projections 
of  a  system  that  is  still  in  the  concept  phase. 

In  addition  to  quantifying  the  probability  of  mission  success  for  each 
proposed  solution,  the  MAJCOM  will  be  developing  mission-level  measures  to 
address  the  COEA  objectives.  As  the  S3rstem  progresses  through  the  acquisi¬ 
tion  process  these  high-level  mission  success  MOEs  should  not  change.  In 
most  cases  the  data  used  to  address  these  MOEs  will  be  aggregated  from 
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lower-level  MOPs  that  can  be  supported  by  actual  test  data.  As  the  EC 
system  proceeds  throu^  the  acquisition  process,  the  data  used  to  evaluate 
the  MOPs  and  the  way  the  data  is  collected  will  change  as  more  complex 
simulations  are  run,  to  testing  the  actual  system  in  hybrid  GTFs,  and  to 
eventually  testing  the  system  in  the  field. 

It  is  also  during  the  concept  exploration  phase  that  the  MAJCOM  prepares 
the  initial  system  threat  assessment.  Here  the  threats  to  the  proposed  system 
concept  are  assessed  and  documented  in  the  system  threat  assessment  report 
(STAR).  By  tailoring  the  STAR  to  the  specific  system  concept,  the  assessment 
identifies  any  potential  or  projected  capabilities  that  the  enemy  cuuld  use  to 
defeat,  destroy,  degrade,  or  deny  the  effectiveness  of  the  proposed  system.^ 
The  threat  information  documented  in  the  STAR  will  be  validated  by  the 
appropriate  intelligence  agency  and  will  be  made  available  to  the  milestone 
decision  authority  prior  to  each  milestone  decision,  beginning  with  mile¬ 
stone  I. 

At  the  end  of  the  concept  exploration  phase,  the  results  are  reviewed  and 
approved  by  the  program  decision  authority.  The  review  will  confirm  that  the 
study  supports  the  need  for  a  new  program  and  that  the  threat  assessment 
has  been  validated.  The  decision  authority  will  also  ensure  that  the  EC  sys¬ 
tem  requirements  have  been  refined,  an  initial  assessment  of  the  cost  and 
development  risk  has  been  made,  and  adequate  test  resources  can  be  made 
available  to  support  the  test  program.  Finally,  the  decision  authority  will 
want  to  see  that  a  test  plan,  with  exit  criteria,  has  been  developed  for  the 
demonstration  and  validation  (dem/val)  phase. 

Demonstration  and  Validation 


The  aim  of  the  demonstration  and  validation  phase  is  to  conduct  technology 
demonstrations  and  to  build  and  test  prototypes  of  the  EC  system.  The 
developing  contractor  may  develop  a  detailed  version  of  the  EC  digital  system 
model  for  use  in  making  performance  predictions  in  the  proposed  mission 
scenario.  These  predictions  will  be  used  to  refine  the  design  of  the  EC  system. 
The  contractor  will  also  test  an  engineering  prototype  of  the  EC  system  to 
assess  the  technologies  needed  to  support  their  proposed  solution  and  design 
concept. 

During  dem/val,  the  developing  contractor  will  focus  his  testing  on  identify¬ 
ing  and  solving  design  risks  and  demonstrating  the  resolution  of  those  risks. 
To  support  developmental  testing,  the  contractor  can  build  an  engineering 
prototype  of  the  preferred  EC  system  by  replacing  certain  components  in  the 
EC  digital  system  model  with  hardware  components  developed  during 
dem/val. 

This  engineering  protot3rpe  will  usually  be  integrated  with  a  computer 
simulation  of  the  host  aircraft  and  the  functional  capabilities  of  the  onboard 
avionics  systems.  The  engineering  prototype  can  then  be  tested  in  the 
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contractor’s  system  integration  laboratory  (SIL)  before  any  prototype  com¬ 
ponents  are  tested  in  government  test  facilities.  Contractor  testing  will  focus 
on  identifying  hardware  and  software  problems,  and  refining  system  perfor¬ 
mance. 

Once  the  contractor  has  finished  building  and  testing  the  EC  digital  system 
model  or  engineering  prototypes,  the  SPO  is  invited  to  participate  in  or  to 
monitor  further  contractor  testing.  Data  is  generated  to  make  performance 
trade-offs  and  to  identify  the  preferred  design.  The  SPO  can  then  study 
performance  predictions  from  the  engineering  protot3rpe  to  ensure  that  the 
proposed  system  design  meets  the  needs  of  the  user  and  validates  the 
proposed  system  concept.  The  SPO  can  also  use  performance  predictions  to 
assess  the  operational  effectiveness  and  suitability  aspects  of  the  design. 

The  SPO  will  be  directing  most  of  its  attention  to  assessing  the  performance 
of  the  components,  subsystems,  or  engineering  protot3rpe8.  The  information 
that  is  gathered  will  be  used  to  verify  that  performance  is  as  predicted,  that 
the  proposed  design  can  indeed  meet  the  user’s  requirements,  and  that  tech¬ 
nologies  critical  to  the  design  can  be  incorporated  into  the  system  at  accept¬ 
able  risk.  The  SPO  will  be  using  engineering  assessments  and  computer 
simulations  to  assist  in  setting  performance  requirements  and  objectives  for 
technical  performance  goals. 

Results  from  computer  simulations  will  be  used  to  demonstrate  that  mile¬ 
stone  exit  criteria  have  been  achieved  and  to  establish  performance  criteria 
and  objectives  for  milestone  III.  Computer  HimulationH  will  also  be  used  to 
determine  certain  mission-level  measures  that  cannot  be  directly  measured  in 
the  hybrid  GTFs  or  the  field  environment. 

Reliability  and  maintainability  (R&M)  data  will  also  be  collected  and 
evaluated  to  make  stire  the  reliability  growth  plans  meet  the  R&M  goals.  'The 
SPO  will  be  defining  and  integrating  electronic  warfare  vulnerability  assess¬ 
ment  (EWVA)  objectives  into  the  DT&E  plans  to  assess  system  vulnerabilities 
and  susceptibilities  to  the  threat  environment.  Furthermore,  the  SPO  will 
form  and  chair  a  test  planning  working  group  (TPWG)  that  will  serve  as  a 
medium  to  discuss  the  development  of  the  test  and  evaluation  master  plan 
with  the  appropriate  test  agencies  and  contractors. 

Once  computer  predictions  of  the  system’s  performance  are  made,  the 
developing  contractor  or  the  SPO  may  also  want  to  test  the  prototype  in  a 
hybrid  GTF  to  assess  its  performance  against  man-in-the-loop  threat 
simulators.  Results  from  this  testing  can  be  used  to  verify  certain  design 
specifications,  calibrate  models,  or  substantiate  pretest  performance  predic¬ 
tions.  The  SPO  may  also  take  the  prototype  system  to  a  measurement  facility 
where  data  can  be  gathered  on  antenna  patterns  and  on  the  host  aircraft’s 
RCS  and  IR  signatures.  This  information  is  needed  for  use  in  computer 
simulations  and  the  hybrid  GTFs. 

Usually  any  field  testing  of  a  prototype  system  during  dem/val  can  only  be 
accomplished  in  an  airborne  laboratory  or  test  aircraft.  Testing  the  prototype 
in  the  field  environment  car  subject  the  system  to  phenomena  that  can  only 
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be  investigated  under  actual  flight  conditions.  Results  from  testing  in  the 
field  may  lead  to  changes  in  system  specifications  or  may  result  in  a  redesign 
of  the  system.  This  is  also  a  good  time  to  again  ensure  that  any  new  test 
resources  or  upgrades  to  existing  test  facilities  are  identified  and  made  ready 
when  full-scale  testing  begins.  The  product  from  the  SPO’s  efforts  in  the 
dem/val  phase  is  a  set  of  documents  that  contains  detailed  cost,  schedule,  and 
system  performance  objectives  and  parameters. 

During  dem/val,  the  OTA  will  continue  the  EOA  activities  by  providing  the 
decision  makers  with  an  assessment  of  the  EC  system’s  potential  to  meet  the 
user’s  mission  requirements.  Problems  uncovered  during  the  EOA  that  could 
impact  the  system’s  operational  capability  must  be  identified  and  rated  so  the 
decision  maker  can  assess  any  high-risk  areas.  Early  involvement  by  the 
OTA  can  provide  the  opportunity  to  influence  the  system’s  development  and 
to  make  sure  the  user’s  needs  are  not  overcome  by  schedule  or  cost  motiva¬ 
tions  in  the  push  to  produce  a  system. 

Initial  EOA  activity  will  involve  a  thorough  review  of  operational  require¬ 
ments,  critical  operational  issues,  and  the  program  documentation  (e.g.,  PMD, 
MNS,  ORD,  STAR,  and  COEIA).^  The  critical  operational  issues  will  be 
reviewed  and  refined  to  verify  that  the  test  plan  addresses  the  required  opera¬ 
tional  characteristics.  In  assessing  the  status  of  the  documentation,  the  OTA 
will  look  for  completeness,  clarity,  and  consistency  between  documents;  suffi¬ 
cient  and  rational  user  requirements;  and  other  factors  that  could  affect 
testability. 

The  OTA  must  work  with  the  user  to  ensure  the  evaluation  criteria  are 
complete  and  testable.  User  evaluation  criteria  should  be  specified  for  both 
the  mission-  and  system-level  measures.  The  evaluation  criteria  should  also 
be  documented  prior  to  the  start  of  OT&E.  'The  OTA  will  assess  the  test 
schedule  to  see  if  it  is  adequate  in  terms  of  time  and  that  the  availability  of 
test  articles  is  sufficient  to  meet  the  needs  of  OT&E  objectives. 

In  this  phase,  the  OTA  will  be  calling  on  the  intelligence  community  to 

*  provide  DIA-validated  intelligence  support  to  the  models  and  simula¬ 
tions, 

•  provide  analysis  of  the  projected  threat  environment, 

*  determine  the  most  realistic  threat  layout  for  the  simulators  in  the 
hybrid  GTF  or  on  the  test  range, 

•  provide  support  in  developing  and  validating  the  threat  scenario, 

•  supply  technical  data  on  the  threats  of  interest,  and 

*  assist  in  range  improvement  programs.^ 

Once  the  threats  have  been  identified  and  tailored  to  the  proposed  system 
concept,  the  OTA  will  determine  availability  of  threat  simulators  to  replicate, 
to  the  best  of  their  ability,  the  threat  environment  in  either  the  hybrid  GTFs 
or  in  the  field. 

In  addition  to  the  threa  t  resources,  the  OTA  will  assess  the  availability  of 
other  test  resources  needed  for  the  evaluation  and  document  any  shortfalls.  A 
test  program  outline  (TPO)  will  be  prepared  as  a  resource  management  and 


programming  plan  to  ensure  resource  requirements  are  programmed  into  the 
budget  cycle.  Alternate  courses  of  action  will  be  planned  in  the  areas  where 
the  test  resources  will  not  be  available.  The  OTA  will  also  make  sure  that  the 
TEMP  includes  a  description  of  the  test  resources  needed  to  support  the 
operational  evaluation. 

The  early  operational  assessment  will  include  a  review  of  the  operational 
aspects  of  the  proposed  approach  and  will  identify  programmatic  voids  that 
could  impact  the  ability  of  the  system  to  meet  user  requirements.  Along  these 
same  lines,  the  OTA  will  assess  the  progress  of  the  system’s  development  and 
identify  significant  trends  in  the  development  process  that  could  impact  the 
system’s  ability  to  meet  user  requirements. 

During  demAral  there  will  be  few,  if  any,  test  articles  that  can  be  used  to 
support  the  EOA.  Basically,  the  operational  test  agency  is  relegated  to  an 
over-the-shoulder  assessment  of  the  EC  system  and  the  activities  of  the 
development  program.  The  OTA  must  work  with  the  SPO  to  ensure  the 
proposed  system  concept  will  meet  the  user’s  operational  requirements. 
When  gathering  data  to  support  the  EOA,  the  OTA  will  have  to  depend  on  the 
results  from  the  SPO’s  or  contractor’s  M/S  efforts  as  a  source  of  information. 
However,  to  have  confidence  that  the  results  are  not  biased  in  favor  of  the 
contractor,  the  OTA  must  understand  how  the  model  operates  and  whether  it 
has  been  accredited.  Other  sources  of  information  include  monitoring  con¬ 
tractor  testing  and  technology  demonstrations,  reviewing  contractor  technical 
performance  documents,  and  sharing  data  from  prototype  testing  if  available. 

The  milestone  II  review  of  the  dem/val  results  should  provide  confidence 
that  the  technologies  and  the  critical  development  processes  for  the  EC  sys¬ 
tem  are  achievable.  The  review  should  further  provide  assurances  that  the 
threat  assessment  and  mission  need  are  still  current  and  valid.  It  is  at  this 
point  that  the  MAJCOM  will  provide  a  detailed  COEA  that  analyzes  and 
evaluates  a  range  of  alternatives.  This  COEA  should  establish  objectives  that 
point  out  the  minimum  acceptable  performance  and  the  maximum  allowable 
cost,  or  some  combination  of  performance  and  cost,  document  the  com¬ 
promises  used  to  arrive  at  the  performance  and  cost  objectives,  and  analyze 
the  consequences  of  terminating  the  program.^  In  addition,  the  MAJCOM 
will  perform  a  COEA  sensitivity  analysis  to  identify  any  critical  sensitivities 
of  the  EC  system’s  effectiveness  to  test  restrictions,  such  as  safety  constraints 
or  test  resource  limitations.  After  the  program  decision  authority  grants  ap¬ 
proval  to  proceed  into  the  engineering  and  manufacturing  development  phase 
(see  fig.  10),  the  MAJCOM  develops  exit  criteria  for  the  next  phase.  Then  the 
program  decision  approves  the  production  of  prototype  systems  to  provide  test 
articles  for  operational  testing. 

The  objectives  of  the  next  phase  in  the  acquisition  process  are  to  assure 
that  the  design  risks  have  been  solved,  to  verify  the  adequacy  of  the  proposed 
manufacturing  processes,  and  to  provide  realistic  production  costs  and 
schedule  estimates. 
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Engineering  and  Manufacturing  Development 


After  approval  from  the  program  decision  authority  to  proceed,  the  en¬ 
gineering  and  manufacturing  development  phase  begins  by 

*  developing  a  production  prototype, 

*  performing  contractor  testing, 

*  starting  the  initial  low-rate  production, 

*  installing  the  EC  system  in  its  host  aircraft,  and 

*  executing  developmental  and  operational  testing.**^ 

It  is  in  this  phase  that  a  complete  and  fully  ftinctional  EC  system  is  developed 
from  requirements  that  evolved  during  the  dem/val  phase. 

As  described  in  DODI  5000.2,  Defense  Acquisition  Management  Policies  and 
Procedures,  the  pvirpose  of  the  engineering  and  manufacturing  development 
phase  is  to  transform  the  most  promising  design  approach  into  a  firmly  estab¬ 
lished,  producible,  and  cost-effective  EC  system  design;  to  substantiate  the 
manufacturing  or  production  process;  and  to  demonstrate  through  testing 
that  the  contract  specifications  have  been  met  and  that  the  system 
capabilities  satisfy  the  mission  need  and  achieve  the  stated  performance 
goals.^^  Also,  the  reliability  and  maintainability  growth  goals  can  bs  reas¬ 
sessed  by  using  computer  simulations  and  actual  test  results  from  the  produc¬ 
tion  protot3rpe8,  to  ensure  that  R&M  performance  objectives  will  be  satisfied 
at  the  milestone  III  decision. 

During  this  phase,  the  EC  digital  system  model  will  be  updated  and 
calibrated  by  the  SPO  with  production  prototype  test  data.  This  will  give  the 
model  the  ability  to  support  further  testing  by  characterizing  the  system’s  full 
performance  range  and  effectiveness.  The  updated  EC  system  model  can  also 
be  used  to  extend  the  test  results  to  other  test  conditions  and  scenarios,  assess 
proposed  design  changes,  and  provide  data  that  address  the  COEA  objectives. 
The  model  can  also  be  used  to  support  pretest  planning,  to  structure  test 
trials,  and  to  update  the  predicted  system  performance  values. 

In  the  engineering  development  phase,  the  contractor  will  start  with  the 
engineering  prototype  components  developed  during  dem/val  and  as  long  as 
the  components  are  a  sound  design,  the  contractor  can  modify  or  gradually 
build  them  up  into  a  fully  functional  production  prototype  of  the  EC  system 
for  testing  in  the  contractor’s  system  integration  laboratory.  The  production 
protot]q>e  system  can  then  be  interfaced  with  real-time  simulations  or  actual 
hardware  from  the  host  aircraft  and  other  avionic  systems.  ’This  integrated 
configuration  can  then  be  stimulated  by  threat  signals  to  test  and  correct  any 
hardware  and  software  problems.  By  using  the  SIL,  the  contractor  can 
evaluate  the  performance  of  the  production  prototype  and  the  compliance 
with  83rstem  specifications.  Once  the  EC  system  prototype  has  been  tested  at 
the  contractor’s  facilities,  it  can  proceed  to  government  testing  for  a  more 
thorough  evaluation. 
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The  SPO  will  be  directing  its  attention  to  testing  the  production  prototype 
to  ensure  that  the  engineering  is  complete  and  that  the  contractor  has  met 
contractual  specifications.  A  test  plan  developed  by  the  contractor  and  SPO 
will  be  the  means  through  which  this  is  accomphshed.  By  collecting  data  to 
evaluate  the  accomplishment  of  development  objectives,  design  problems  wrill 
be  identified  and  solved,  system  software  tested,  compatibility  and  inter¬ 
operability  with  other  aircraft  systems  checked,  and  R&M  results  examined. 
If  there  have  been  significant  changes  to  the  performance  or  costs  to  the 
system,  the  SPO  will  also  make  sure  that  the  data  needed  by  the  MAJCOM  to 
update  the  COEIA  has  been  collected  and  is  ready  for  the  milestone  III  review. 

The  SPO  begins  the  developmental  test  process  by  using  M/S  to  predict  the 
performance  of  the  EC  s)rstem  under  various  test  conditions  and  to  complete 
any  remaining  pretest  planning  factors.  The  testing  then  proceeds  to  the 
contractor’s  SIL  for  compliance  testing  on  a  production  prototype.  Here  the 
SPO  monitors  tests  that  confirm  that  the  system  performs  as  designed  and 
ensures  that  any  identified  hardware  and  software  problems  are  fixed. 

The  production  protot3rpe  is  then  placed  in  an  uninstalled  hybrid  GTF  to 
confirm  that  problems  identified  in  previous  testing  have  been  corrected  and 
that  the  specified  performance  objectives  for  the  EC  system  have  been 
achieved.  If  the  system  is  designed  to  use  electronic  countermeasures  (ECM) 
or  IR  countermeasures  to  oppose  the  threat  system  then  based  on  the  evalua¬ 
tion  criteria,  the  hybrid  GTF  can  confirm  if  the  countermeasures  used  against 
the  threat  are  effective.  Examples  of  the  type  of  testing  that  can  take  place 
here  include  the  capability  to  process  a  high-density  signal  environment,  to 
determine  system  effectiveness  in  terms  of  inducing  tracking  errors  and 
missile-miss  distances  against  MITL  threat  simulators,  and  to  verify  that 
data  received  from  several  sensor  locations  is  correctly  fused  into  a  single 
output  for  display.  Once  again,  the  SPO  will  have  to  take  the  production 
prototype  to  the  measurement  facilities  to  establish  new  or  updated  values  for 
antenna  patterns  and  aircraft  RCS  and  IR  signatures  for  use  as  inputs  to  the 
simulations  performed  in  the  hybrid  GTF. 

By  testing  in  the  installed  hybrid  GTF,  the  SPO  can  evaluate  the  perfor¬ 
mance  of  the  EC  system  while  the  system  is  installed  on  the  host  aircraft. 
This  may  be  the  first  opportunity  to  test  the  EC  system  when  it  is  installed  on 
its  host  aircraft  and  integrated  writh  onboard  avionics.  Developmental  testing 
is  conducted  in  the  hybrid  GTF  to  evaluate  the  integrated  performance  of  the 
EC  system  as  part  of  the  total  avionics  suite.  Examples  of  installed  testing 
include 

•  electromagnetic  compatibility  (EMC)  with  other  systems  on  the  eiircraft, 

•  sensor  operation, 

•  locating  and  displaying  the  threat  system  to  the  aircrew, 

•  data  flow  to  EC  S3rstem  processors  and  output  di8pla3rs,  and 

•  confirmation  of  RF  isolation  between  the  receive  and  transmit  antennas. 
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The  installed  hybrid  GTF  can  also  identify  major  problems  during  the 
preflight  of  the  system  on  the  day  of  the  flight  and  provide  a  method  to 
analyze  problems  encountered  during  flight. 

Field  testing  follows  ground  testing  and  provides  the  opportunity  for  the 
SPO  to  establish  perfonnance  values  for  the  EC  system  in  an  open-air  en¬ 
vironment.  Once  again,  key  system-performance  parameters  are  tested  in 
representative  mission  scenarios.  They  begin  with  one-versus-one  engage¬ 
ments  and  progress  to  scenarios  with  multiple-threat  engagements  using  all 
available  test  resources.  Examples  of  data  collected  during  flight  testing  in¬ 
clude  countermeasure  effectiveness  against  MITL  threat  simulators,  angle  of 
arrival  accuracies,  threat-detection  ranges,  correct  threat  identification,  and 
data  for  R&M  analysis. 

Because  many  of  the  test  resources  needed  to  support  DT&E  and  OT&E  are 
the  same,  testing  can  be  conducted  in  a  combined  effort,  generating  similar 
test  data  used  to  support  each  other’s  test.  Although  DT&E  and  OT&E  are 
still  separate  and  distinct  test  programs,  the  benefit  of  a  combined  test  pro¬ 
gram  comes  from  the  cost-effective  use  of  test  resources  and  from  reducing  the 
time  to  test  the  EC  system. 

Although  there  are  similarities  between  DT&E  and  OT&E,  there  is  one 
important  difference  that  has  to  be  taken  into  consideration  when  operational 
testing  proceeds  beyond  the  LRIP  decision.  That  is  the  use  of  development 
contractors  in  OT&E.  The  US  Code  allows  the  use  of  development  contractors 
in  all  phases  of  DT&E.  However,  as  pointed  out  in  chapter  3,  Title  10  of  the 
US  Code  prohibits  the  involvement  of  development  contractors  in  OT&E 
when  proceeding  beyond  the  LRIP  decision,  except  to  the  extent  that  DOD 
plans  call  for  their  involvement  in  the  operation,  maintenance,  and  support  of 
the  system  when  deployed.  Because  of  this  restriction,  the  OTA  will  have  to 
be  extremely  cautious  of  contractor  involvement  when  dedicated  lOT&E 
begins. 

Planning  a  combined  test  requires  close  coordination  between  the  SPO  and 
OTA  to  ensure  the  test  conditions  and  data  requirements  necessary  for  both 
test  programs  are  satisfied.  In  the  initial  stages  of  the  test  process,  DT&E 
test  events  will  have  priority  in  evaluating  the  critical  technical  and  engineer¬ 
ing-level  performance  measures.  The  OTA  will  participate  in  this  stage 
mainly  as  an  observer,  while  becoming  familiar  with  the  system  and  identify¬ 
ing  test  data  that  can  be  used  to  support  OT&E.  The  next  segment  of  the  test 
will  include  test  events  which  will  produce  data  that  can  be  shared  between 
the  developmental  and  operational  test  programs.  This  combined  approach 
should  not  interfere  with  either  test  program.  At  the  completion  of  DT&E, 
the  SPO  wUl  certify  that  the  EC  system  is  ready  for  dedicated  OT&E.  The 
last  stage  of  testing  is  then  conducted  by  the  OTA  and  contains  test  events 
dedicated  to  lOT&E.^^ 

After  entering  the  engineering  and  development  phase,  the  OTA  will  begin 
by  making  an  independent  operational  assessment  (OA)  of  the  system’s  poten¬ 
tial  to  meet  the  user’s  requirements  and  its  progress  toward  becoming  an 
operationally  effective  and  suitable  S3r8tem.  The  purpose  of  the  OA  is  to 
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support  the  LRIP  decision  (see  fig.  10).  Again,  the  role  of  the  OTA  during  the 
OA  wiU  mainly  be  that  of  an  observer  monitoring  the  progress  of  DT&E.  The 
OA  will  be  based  on  all  relevant  data  from  developmental  testing,  user  trials, 
interim  results,  and  computer  simulations.  Many  of  the  same  activities  and 
program  reviews  that  took  place  during  the  dem/val  phase  will  be  continued 
by  the  OTA  in  the  operational  assessment.  This  will  include  significant 
developmental  trends  affecting  the  system’s  ability  to  meet  the  mission  need, 
programmatic  voids,  areas  of  risk,  and  adequacy  of  test  requirements,  and  the 
ability  of  the  program  to  support  adequate  operational  testing. 

During  the  OA,  many  of  the  planning  activities  that  go  into  developing  an 
operational  test  plan  are  accomplished.  This  is  also  the  time  to  identify  the 
necessary  test  resources,  to  finalize  the  proper  mix  of  test  tools,  and  to  form 
the  test  teeim. 

If  available  during  the  OA,  the  updated  EC  digital  system  model  can  help 
with  the  pretest  planning  step  for  lOT&E  and  to  msdre  performance  predic¬ 
tions  to  compare  with  test  results  from  the  hybrid  GTF  and  the  field  test.  If 
not  already  established,  the  mission-  and  system-level  measures  are 
developed  or  at  least  refined  along  with  the  essential  data  elements  needed 
for  analysis.  This  includes  the  elements  needed  from  field  testing  to  validate 
and  accredit  the  models.  The  OTA  may  also  be  tasked  to  support  the 
MAJCOM’s  analysis  of  the  system’s  cost  and  performance  trade-offs  by 
providing  data  to  run  in  the  MAJCOM’s  COEA  models.  If  this  is  the  case, 
then  the  test  elements  needed  to  support  this  effort  must  be  included  in  the 
lOT&E  test  plan.  This  initial  use  of  computer  simulation  is  used  to  gain 
insighi  into  the  EC  system’s  response  to  the  threat.  It  is  not  intended  to 
determine  or  evaluate  the  effectiveness  of  the  system. 

Using  the  MAJCOM-developed  computer  simulation  of  the  planned  opera¬ 
tional  scenario,  the  test  manager  can  generate  the  field-test  scenarios  and 
identify  the  data  items  that  must  be  described  in  the  lOT&E  test  plan.  The 
test  manager  must  remember  that  when  using  M/S  in  OT&E  to  support  a 
decision  to  proceed  beyond  the  LRIP,  the  involvement  of  development  contrac¬ 
tors  in  conducting  or  assisting  in  the  test,  or  as  advisors,  is  restricted  by  Title 
10  of  the  US  Code}*  Therefore,  no  development  contractor  involvement  or 
the  use  of  development  contractor  facilities  will  be  relied  upon  in  lOT&E.  It  is 
for  this  reason  that  the  contractor  system  integration  laboratory  will  not  be 
integrated  into  dedicated  lOT&E  test  plans. 

The  operational  test  agency  must  make  sure  everything  is  in  place  and 
ready  for  dedicated  initial  operational  test  and  evaluation.  Then  after  the 
SPO  certifies  the  EC  system  “ready,”  the  OTA  is  cleared  to  conduct  dedicated 
lOT&E  on  a  production-representative  system. 

The  EC  test  process  for  OT&E  looks  very  similar  to  that  of  DT&E,  but  the 
focus  of  operational  testing  is  on  answering  the  COIs  instead  of  verifying 
technical  system  performance.  Furthermore,  the  test  scenarios  used  in  OT&E 
will  be  more  representative  of  the  proposed  operational  missions  than  of  the 
very  controlled  one-versus-one  engagements  conducted  during  DT&E. 
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Once  dedicated  lOT&E  has  started,  it  is  essential  that  a  production- 
representative  system  be  used  because  it  provides  the  most  vahd  test  ap¬ 
proach  to  evaluate  operational  effectiveness  and  suitability  objectives.  Then, 
through  the  application  of  a  sound  EC  test  process,  the  COls  refined  during 
the  dem/val  phase  can  be  addressed,  and  the  OTA  can  ensure  that  an  opera¬ 
tionally  effective  and  suitable  EC  system  is  delivered  to  the  using  command. 

The  OTA  may  initially  take  the  production-representative  system  to  an 
uninstalled  hybrid  GTF  where  data  will  be  collected  to  answer  operational 
effectiveness  and  suitability  objectives.  The  type  of  test  data  that  is  to  be 
collected  will  depend  on  the  measures,  but  in  general  it  will  be  similar  to  that 
collected  in  DT&E.  It  is  the  testing  philosophy  that  differs  from  developmen¬ 
tal  testing  in  that  the  scenarios  are  validated  and  the  threat  densities  used  for 
the  evaluation  are  operationally  representative.  In  addition,  the  OTA  can  use 
the  hybrid  GTFs  to  complement  field  testing  by  providing  an  opportunity  to 
pretest  the  mission  scenario  before  testing  in  the  field.  This  pretest  will  allow 
the  OTA  to  further  refine  the  test  plan  and  make  more  efficient  use  of  the 
available  field  range  time.  Capt  William  Parmer  and  Col  John  Nagel  further 
state  that 

the  test  director  can  identify  sensitive  areas  which  will  'equire  special  attention  in 
the  field,  refine  instrumentation  requirements,  revise  tactics,  and  identify  the  im¬ 
pact  of  the  human  threat  system  operator  on  the  effectiveness  of  the  EW  (electronic 
warfare)  system  under  test.  “ 

Next  the  OTA  can  test  the  EC  system  in  an  installed  hybrid  GTF  where  the 
system’s  performance  can  be  evaluated  while  integrated  with  the  host 
aircraft’s  avionic  systems.  The  testing  methods  in  this  facility  are  similar  to 
those  used  in  DT&E.  The  exception  is  that  the  threat  environment  is  more 
representative  of  an  operational  environment,  and  the  focus  of  the  test  is  to 
address  the  COIs. 

Field  testing  gives  the  OTA  the  opportunity  to  evaluate  the  EC  system  with 
its  host  aircraft  under  natural  environmental  operating  conditions.  Real- 
world  phenomena  such  as  terrain  effects,  multipath  propagation,  and  com¬ 
mercial  electromagnetic  interference  effects  (radio  broadcasts,  microwave 
transmissions,  etc.)  can  only  be  encountered  in  the  field  environment.  Inter¬ 
ference  and  system  incompatibilities  due  to  these  circumstances  will  usually 
show  up  in  the  field  test  and  not  in  the  computer  simulations  or  the  hybrid 
GTFs.  Field  testing  provides  the  OTA  the  opportunity  to  see  how  the 
operator,  functioning  under  the  pressures  of  an  actual  flight  environment,  can 
operate  the  EC  system  and  interpret  the  system’s  output. 

It  is  during  field  testing  that  the  OTA  can  adequately  address  system 
suitabiUty  in  a  reaUstic  environment.  This  area  cannot  be  addressed  suffi¬ 
ciently  with  computer  simulations  or  in  a  ground  test  facihty.  Also,  if  the  EC 
system  is  designed  to  detect  the  presence  of  missiles  fired  at  the  aircraft,  then 
field  testing  can  provide  an  environment  to  conduct  live  firings  of  missiles  at 
the  aircraft,  from  a  safe  distance,  to  test  the  complete  end-to-end  performance 
of  the  EC  system.  Once  again  testing  in  the  field  is  quite  similar  to  the 
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testing  accomplished  during  DT&E  with  the  main  difference  being  a  more 
operationally  representative  test  scenario  with  the  focus  on  addressing 
mission-level  test  objectives.  Any  system  problems  encountered  during  test¬ 
ing  will  be  reported  to  the  SPO  for  resolution  at  the  conclusion  of  testing. 

Prior  to  the  milestone  III  review,  the  MAJCOM  will  be  reviewing  the  COEA 
using  test  results  from  both  developmental  and  operational  testing  to  confirm 
the  decision  that  the  selected  system  is  still  the  most  cost-effective  approach 
to  satisfying  the  operational  requirement.^^  Any  analysis  done  to  support  the 
COEA  process  will  only  be  used  to  update  the  current  COEA.  Unless  there 
are  sufficient  changes  to  the  performance  and  cost  estimates,  a  new  cost  and 
operational  effectiveness  assessment  is  not  required.  If  it  is  discovered  during 
the  premilestone  planning  process  that  changes  justify  a  new  COEA,  the 
milestone  decision  authority  will  then  explicitly  state  the  analysis  that  needs 
to  be  accomplished  to  update  the  elements  in  the  COEA. 

At  the  milestone  III  decision,  test  results  will  be  reported  so  that  a  decision 
can  be  made  as  to  whether  to  produce  the  EC  S3r8tem.  Decision  makers  will  be 
using  the  test  results  from  both  developmental  and  operational  testing  in 
making  their  appraisal.  They  will  also  be  verifying  that  the  proposed  design 
and  manufacturing  processes  are  stable,  realistic  estimates  of  production 
costs  are  established,  and  a  plan  to  deploy  and  support  the  system  is  com¬ 
plete.  Additionally,  the  decision  authorities  will  want  to  see  that  the  final 
cost,  schedule,  and  performance  objectives  for  production  and  deplo5rment  are 
documented.  With  a  favorable  decision  by  the  program  decision  authorities, 
approval  is  given  to  enter  the  production  and  deployment  phase  (see  fig.  10). 


Production  and  Deployment 

After  reaching  the  production  and  deployment  phase,  the  EC  system  enters 
into  full-rate  production.  It  is  during  this  phase  that  the  using  command  will 
declare  the  system  ready  for  initial  operational  capability  (IOC).  IOC  is  based 
on  criteria  defined  during  the  previous  phase.  The  full-rate  production  sys¬ 
tems  will  be  the  ones  deployed  to  operational  units  and  used  in  the  next  stage 
ofOT&E. 

The  objective  of  the  production  and  deployment  phase  is  to  make  sure  an 
efficient  and  stable  production  process  is  established  with  an  adequate  techni¬ 
cal  support  base.  In  addition,  the  user  must  be  satisfied  that  the  intended 
mission  need  has  been  met  with  the  capabilities  of  the  system.  If  a  new 
requirement  is  identified  during  this  phas ,  and  it  requires  a  major  modifica¬ 
tion  to  the  system,  then  an  additional  milestone  (milestone  IV)  is  necessary. 
System  modifications  may  be  necessary  to  correct  deficiencies;  to  make  im¬ 
provements  to  the  system;  to  compensate  for  new  intelligence  on  the  threat;  to 
reduce  life-cycle  costs;  or  to  improve  system  reliability,  maintainability,  and 
availability.**  To  confirm  any  of  the  modifications  and  to  substantiate  that 
system  specifications  have  been  met,  DT&E  will  be  conducted.  Developmen- 


66 


tal  testing  may  abo  be  required  to  demonstrate  an  operational  capability  or  to 
identify  requirements  for  additional  operational  testing. 

DT&E  will  evaluate  and  verify  any  changes  in  the  hardware  and  software 
through  a  limited  number  of  installed  ground  tests  and  flight  tests.  Again, 
the  SPO  will  begin  the  EC  test  process  with  pretest  planning  to  evaluate  the 
changes  to  the  system.  Testing  will  then  proceed  through  the  various  test 
tools  as  required  to  support  program  decisions  and  to  update  the  EC  digital 
system  model  so  it  exhibits  the  characteristics  of  the  production  system. 

If  there  are  mqjor  changes  to  the  design  and  function  of  the  system,  then 
the  SPO  will  have  to  verify  that  the  EC  system  is  still  effective  against  MITL 
threat  systems.  The  purpose  for  conducting  DT&E  on  the  modified  produc¬ 
tion  system  is  to  ensure  that  the  performance  and  effectiveness  requirements 
established  for  the  system  can  still  be  achieved.  This  testing  will  hrst  be 
accomplished  at  a  hybrid  GTF,  then  proceed  to  an  installed  hybrid  GTF  where 
the  system’s  interfaces  and  interoperability  requirements  with  the  host 
aircraft’s  avionic  systems  can  be  evaluated.  The  installed  hybrid  GTF  will 
also  be  used  to  pretest  the  system  prior  to  any  field  testing.  Again,  measure¬ 
ment  facility  testing  may  have  to  take  place  if  there  were  changes  to  the 
antenna’s  shape  or  function  or  its  location  on  the  host  aircraft. 

Next,  the  SPO  wrill  perform  the  required  fteld  testing  with  the  EC  system  to 
make  certain  the  production  system  satisfies  the  user’s  requirement.  The  Air 
Force  E’ectronic  Combat  Development  Test  Process  Guide  states  that  “this 
testing  may  include  new  threat  simulators  that  are  more  representative  of  the 
actual  threat  and  may  employ  larger  test  scenarios  as  more  test  assets  are 
available.”^® 

If  any  operational  testing  is  required  during  the  production  and  deployment 
phase,  it  is  referred  to  as  follow-on  operational  test  and  evaluation.  Testing  is 
conducted  because  the  proposed  mission  or  threat  environment  may  have 
changed  to  the  point  where  it  is  necessary  to  see  if  the  system  is  still  effective. 
FOT&E  is  used  to  evaluate  any  modification  or  design  changes  made  to  the 
production  system  as  the  result  of  deficiencies  found  during  lOT&E  and  to 
ensure  it  continues  to  meet  the  user’s  operational  needs.  The  EC  system’s 
operational  effectiveness  and  suitability  objectives  are  further  evaluated  to 
identify  any  system  deficiencies  or  needed  modifications.  By  increasing  the 
number  of  test  events  and  test  conditions,  FOT&E  can  also  be  used  to  refine 
the  operational  effectiveness  and  suitability  results  obtained  during  lOT&E. 

In  this  phase,  the  OTA  will  tailor  the  FOT&E  to  examine  those  areas  that 
are  required  for  a  milestone  IV  review.  As  in  the  previous  phase  of  testing, 
the  OTA  wall  start  the  EC  test  process  with  pretest  planning  to  determine  the 
objectives  of  the  test  and  the  proper  mix  of  test  tools.  Depending  on  the 
objective  of  the  test,  the  OTA  may  take  the  EC  system  to  a  hybrid  ground  test 
facility  to  evaluate  its  effectiveness  against  an  updated  threat  environment  or 
may  just  take  it  into  the  field  to  reaccomplish  portions  of  the  previous  evalua¬ 
tion.  In  any  case,  the  testing  performed  in  FOT&E  wall  be  comparable  to  that 
conducted  in  lOT&E  and  will  proceed  in  a  similar  manner. 
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The  purpose  of  the  milestone  IV  decision  is  to  give  approval  to  initiate  a 
major  modiAcation  (see  fig.  10).  If  approval  is  given  to  an  upgrade  or 

modification  to  a  system  that  is  in  production,  then  the  decision  authority 
may  require  a  COEA.  If  a  COEA  is  required,  then  t  .e  decision  authority  will 
specify  the  elements  that  will  require  further  analysis.  As  part  of  the  pro¬ 
gram  review,  the  decision  authority  will  make  sure  that  all  alternatives  to  the 
modification  have  been  considered.  They  will  also  confirm  that  the  perfor¬ 
mance  objectives  have  been  met  and  any  new  assessment  of  the  threat  has 
been  validated.  The  decision  makers  will  be  basing  part  of  their  conclusions 
on  performance  of  the  system  in  the  operational  units  as  well  as  the  results 
from  testing.  They  will  be  examining  and  determining  if  the  required  tech¬ 
nologies  and  production  processes  have  been  clearly  identified  and  can  be 
attained.  Finally,  the  program  decision  authority  will  re  .lew  the  affordability 
of  the  program  and  the  availability  of  the  resources  needed  to  support  the 
program. 

Operations  and  Support 

Once  the  EC  system  is  fully  operational,  it  enters  the  operations  and  sup¬ 
port  phase  of  the  acquisition  process.  There  will  be  some  overlap  between  this 
phase  and  the  production  and  deplo3rment  phase  (see  fi(  10).  The  operations 
and  support  phase  signals  the  transfer  of  the  memagement  responsibilities 
from  the  SPO  to  the  system  program  manager  at  the  logistic  center.  The  EC 
system  remains  in  this  phase  until  it  is  retired  from  the  inventory. 

The  objective  of  testing  in  this  phase  is  to  ensure  that  the  EC  system 
continues  to  meet  the  user’s  mission  requirements  and  to  identify  any 
deficiencies  or  modifications  needed  to  improve  the  performance  of  the  sys¬ 
tem.  To  confirm  the  performance  improvements  and  evaluate  the  system’s 
operational  effectiveness,  FOT&E  may  have  to  be  conducted  with  several  of 
the  T&E  tools.  The  EC  test  process  will  again  be  used  to  plan,  conduct,  and 
report  the  test  results  to  the  decision  authorities. 

This  concludes  the  description  of  the  tasks  to  be  accomplished  in  each  phase 
of  the  acquisition  process  and  where  the  T&E  tools  fit  to  support  the  process. 
As  this  chapter  has  shown,  OT&E  is  an  iterative  process  that  overlays  the 
developmental  test  process  with  a  mission-level  evaluation.  The  respon¬ 
sibilities  of  the  participating,  operating,  and  supporting  commands,  and  the 
OTAs  must  be  integrated  into  a  coherent  test  process  that  supports  the  ac¬ 
quisition  strategy.  The  test  and  evaluation  master  plan  is  the  document  that 
integrates  all  T&E  and  makes  certain  that  the  test  program  is  consistent  with 
the  acquisition  process. 

The  TEMP  furnishes  the  linkage  between  the  test  schedule  and  program 
schedule.  It  provides  the  basic  structure,  test  philosophy,  and  direction  that 
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go  into  putting  together  a  comprehensive  OT&E  test  plan.^®  The  TEMP 
should  be  updated  annually  to  reflect  any  changes  to  the  requirements, 
schedule,  T&E  tools,  and  so  forth,  and  reviewed  prior  to  each  milestone 
decision  point.^’  The  test  manager  can  use  the  TEMP  as  an  instrument  to 
document  and  lay  out  the  requirements  for  OT&E.  Before  the  TEMP  is  ap¬ 
proved  and  signed,  there  must  be  concurrence  and  agreement  with  the 
schedule,  funding,  testing,  resources,  and  so  forth  between  all  agencies  in¬ 
volved  with  the  T&E  of  the  EC  system.  Once  signed  the  TEMP  becomes  the 
road  map  that  key  decision  makers  will  use  when  following  the  testing  to  be 
accomplished  in  each  phase  of  the  acquisition  process.  As  the  system  matures 
and  various  stages  of  OT&E  are  completed,  the  test  manager  will  have  to 
ensure  the  OT&E  portions  in  the  TEMP  are  updated  and  that  all  changes  to 
the  TEMP  support  the  EC  test  process.  The  test  manager  must  be  particu¬ 
larly  cautious  that  changes  to  the  program  schedule  do  not  destroy  the  test 
process. 

Unless  all  organizations  involved  in  the  EC  test  process  complete  their 
assigned  responsibilities  the  EC  test  process  will  not  work.  For  example,  the 
OTA  cannot  derive  the  operational  test  requirements  until  the  MA.JCOM  has 
defined  the  operational  scenario  and  the  mission  need. 

Finally,  it  should  be  noted  that  the  test  manager  does  not  have  to  use  every 
T&E  tool  described  in  this  study.  Depending  on  the  objectives  of  the  test  and 
the  status  of  the  EC  system,  the  test  manager  will  select  just  the  test  tools 
that  are  needed  to  satisfy  the  objectives.  For  example,  during  FOT&E  the 
test  manager  may  not  need  to  go  to  the  uninstalled  hybrid  GTF  if  the  system 
is  already  installed  in  the  aircraft  for  ground  testing.  Or  perhaps  testing  on 
the  field  test  range  may  be  all  that  is  needed  to  address  the  test  objectives. 
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Chapter  5 


Summary,  Conclusions,  and 
Recommendations 


The  introduction  of  EC  systems  into  the  force  structure  has  been  due  in 
part  to  a  reaction  to  the  enemy’s  defensive  systems.  Initially,  EC  systems 
were  developed  under  a  “quick-reaction”  program  that  attempted  to  satisfy 
the  immediate  need  of  countering  the  enemy’s  threat  systems.  Because  of  the 
urgency  to  find  countermeasures,  any  T&E  that  took  place  ended  up  being 
more  of  a  “trial-and-error”  process  that  attempted  to  identify  the  best  method 
to  counter  the  threat.  The  T&E  tools,  such  as  those  described  in  this  study, 
either  did  not  exist  or  were  not  representative  of  the  actual  threat  environ¬ 
ment  and  therefore  were  of  little  use  in  evaluating  the  effectiveness  of  the  EC 
system.  As  a  result,  the  T&E  philosophy  that  evolved  has  been  haphazard 
and  has  lacked  any  resemblance  to  a  standardized  test  process.  What  has 
evolved  and  is  still  being  practiced  is  a  test  process  that  follows  a  “test-fiz- 
test”  method.  This  works  well  in  supporting  the  development  of  the  EC  sys¬ 
tem  but  does  not  support  the  intent  of  OT&E. 

When  several  EC  S3^tems  failed  to  meet  the  user’s  requirements,  a  broad 
area  review  was  initiated  to  determine  if  there  were  deficiencies  in  the  EC 
test  process.  One  of  the  findings  fiom  the  review  stated  that  EC  system 
testing  lacked  the  discipline  and  essential  elements  of  a  scientific  test  process. 
Furthermore,  the  broad  area  review  determined  that  due  to  the  lack  of  a 
disciplined  and  standardized  test  process,  meaningful  test  results  necessary 
to  support  production  decisions  were  not  being  produced.  In  addition,  the 
Department  of  Defense  inspector  general  reviewed  several  OT&E  programs 
and  concluded  that  OT&E  would  have  more  of  an  impact  on  acquisition 
decisions  if  it  did  not  get  caught  up  in  the  test-fiz-test  scenario  that  began  in 
developmental  testing. 

I  have  found  that  to  add  discipline  and  structure  to  the  EC  test  process, 
there  are  certain  limitations  or  challenges  that  must  be  addressed  in  order  for 
the  decision  makers  to  be  satisfied  with  the  OT&E  results.  The  first  limita¬ 
tion  has  to  do  with  testing  the  EC  system  in  an  operationally  representative 
test  environment.  A  test  method  must  be  devised  to  evaluate  the  EC  system 
in  a  test  environment  that  accurately  represents  the  actual  ‘Combat  environ¬ 
ment.  This  will  give  decision  makers  confidence  in  the  test  results  when 
making  a  decision  on  Aether  to  continue  or  terminate  the  program. 
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Decision  makers  want  to  know  not  only  that  the  system  meets  the  technical 
performance  requirements  but  that  it  contributes  to  the  overall  success  of  the 
mission.  Determining  whether  the  system  is  effective  and  plays  a  significant 
part  in  the  success  of  the  mission  is  the  purpose  of  OT&E.  However,  estab¬ 
lishing  mission-level  test  measures  and  criteria  that  evaluate  the  EC  system's 
contribution  to  the  mission  have  not  been  well  defined  or  standardized. 
Mission-level  measures  are  important  to  decision  makers  when  considering 
the  cost  and  effectiveness  of  one  alternative  over  another. 

Another  challenge  to  the  T&E  of  EC  systems  is  the  absence  of  complete 
intelligence  on  the  latest  generation  of  threat  systems.  The  lack  of  complete 
intelligence  has  caused  two  problems  for  the  OTAs.  Both  have  to  do  with  the 
technical  details  that  describe  the  threat  and  its  operation.  First  of  all,  this 
information  is  needed  to  build  and  validate  the  models  and  threat  simulators. 
Without  accurate  threat  simulators,  a  true  assessment  of  the  EC  system’s 
performance  cannot  be  obtained.  Second,  this  information  is  needed  when 
designing  an  EC  system  that  can  identify  and  counter  the  threat  system.  If 
the  technical  details  on  the  threat  are  unknown,  then  the  design  of  the  EC 
system  cannot  be  finalized.  If  it  is  finalized  before  the  necessary  technical 
information  on  the  threat  is  known,  the  EC  system  may  be  ineffective  in 
meeting  the  user’s  requirements.  As  a  result,  the  developer  is  often  chasing 
down  the  latest  information  on  the  threat  to  incorporate  into  the  design  of  the 
EC  system.  Furthermore,  as  new  intelligence  on  the  threat  is  received  during 
developmental  testing,  the  SPO  will  be  attempting  to  integrate  this  informa¬ 
tion  into  the  EC  system  by  updating  the  hardware  and  software  functions. 
Inevitably,  the  OTA  ends  up  testing  a  system  that  is  not  quite  representative 
of  the  production  version  and  may  have  to  test  a  system  that  has  known 
limitations  or  shortfalls  in  meeting  the  user’s  requirements.  When  this  hap¬ 
pens,  the  OTAs  wind  up  providing  test  results  to  the  decision  authorities  that 
are  not  representative  of  the  fmal  system. 

Finally,  the  OTAs  are  faced  with  the  challenge  of  testing  a  system  that  is 
highly  integrated  and  dependent  on  other  onboard  avionic  systems.  'The  in¬ 
tegrated  nature  of  the  total  avionics  suite  makes  it  difficult  to  assess  the 
effectiveness  of  just  the  EC  functions  in  contributing  to  the  success  of  the 
mission.  Furthermore,  because  of  the  dependence  on  other  onboard  avionics, 
the  test  scenario  becomes  more  complicated,  requiring  both  friendly  and  hos¬ 
tile  players  along  with  the  associated  C^  network.  To  provide  structure  and 
discipline  to  the  EC  test  process,  these  limitations  and  challenges  must  be 
addressed. 


Conclusions 


The  solution  rests  with  the  proper  application  of  the  available  T&E  tools 
and  scientific  test  methods.  A  scientific  test  process  will  provide  the  structure 
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and  discipline  that  will  produce  the  information  that  the  decision  makers  are 
looking  for  to  decide  if  the  program  should  proceed  to  the  next  phase  of  the 
acquisition  process.  The  test  results  help  provide  the  knowledge  to  determine 
whether  the  EC  system  is  meeting  its  performance  objectives,  and  whether  it 
is  still  an  affordable  solution.  The  scientific  test  process  will  give  the  decision 
makers  the  confidence  that  the  test  results  reflect  the  way  the  system  will 
work  if  deployed.  In  addition,  it  will  help  the  OTAs  avoid  the  test-fix-test 
scenario  started  in  DT&E. 

A  well-defined  test  process  starts  with  the  selection  of  T&E  tools  needed  to 
answer  the  test  objectives.  There  is  a  wide  range  of  tools  that  can  be  used  in 
the  EC  test  process,  each  tool  having  its  own  particular  purpose  in  supporting 
the  test  program.  The  test  manager  must  understand  the  functional  relation¬ 
ships  between  the  tools  and  how  they  support  the  EC  test  process.  Further¬ 
more,  he  or  she  must  ensure  that  the  tools  are  available  at  the  start  of  each 
stage  of  OT&E  and  that  they  have  been  vahdated  and  accredited  to  provide 
credible  test  results  for  the  evaluation.  This  study  has  identified  three  major 
categories  of  tools  from  which  to  select:  modeling  and  simulations,  hybrid 
ground  test  facilities,  and  field  test  ranges. 

To  add  disciphne  and  structure  to  the  EC  test  process  the  test  manager 
must  apply  the  principles  of  a  scientific  test  methodology  to  the  operational 
evaluation.  There  are  six  fundamental  steps  (see  fig.  7)  to  the  scientific  test 
process  that  can  be  appUed  to  any  EC  test  program.  The  test  methodology 
can  then  be  tailored  in  each  stage  of  OT&E  to  support  the  decision  makers 
vsrith  information  at  the  appropriate  milestone  decision  points  (see  fig.  10). 
The  objective  of  the  scientific  test  process  is  to  facihtate  effective  test  plan¬ 
ning,  test  execution,  and  analysis  of  the  test  results.  Furthermore,  the  EC 
test  process  depicted  in  figure  6  stresses  the  integrated  use  of  M/S,  hybrid 
GTFs,  and  field  testing  to  evaluate  the  operational  effectiveness  and 
suitability  objectives.  The  result  is  a  disciplined  EC  test  process  that  will 
produce  the  products  needed  to  conduct  an  orderly  OT&E.  Also,  by  identify¬ 
ing  the  proper  functional  relationships  between  the  T&E  tools  and  emphasiz¬ 
ing  their  role  in  the  EC  test  process,  many  of  the  hmitations  or  challenges  to 
operational  testing  can  be  dealt  with. 

By  implementing  a  scientific  test  process,  the  OTAs  will  have  a  much  better 
idea  of  what  to  expect  during  the  performance  of  OT&E.  The  scientific  test 
process  uses  a  methodology  that  forms  pretest  predictions  that  will  give  the 
OTAs  an  indication  of  what  to  expect  before  the  test  starts.  Knowing  what  to 
expect  from  the  EC  system  ahead  of  time  will  allow  better  preparation  for  the 
evaluation  by  the  OTAs. 

The  test  manager  must  identify  the  aim  of  OT&E  in  each  phase  of  the 
acquisition  process  and  focus  in  on  where  the  T&E  tools  fit  into  the  scheme  of 
OT&E.  The  current  acquisition  process  does  provide  the  framework  in  which 
to  base  a  structured  test  program.  The  five  phases  and  milestone  decision 


points  in  the  acquisition  process  are  designed  to  make  an  orderly  transition  of 
broadly  stated  mission  needs  into  system-specific  performance  requirements. 
Within  each  phase,  the  OTAs  play  an  important  part  in  estimating  whether 
the  proposed  solution  will  meet  the  user’s  requirements  by  providing  decision 
makers  with  information  on  the  system’s  effectiveness  and  suitability.  By 
implementing  a  scientific  test  process,  OT&E  will  then  become  an  excellent 
vehicle  to  make  significant  contributions  to  the  acquisition  process  and  sup¬ 
port  the  intent  of  the  decision  makers. 

However,  much  of  what  needs  to  be  done  is  outside  the  purview  of  the 
OTA’s  scope  and  authority.  Other  than  implementing  a  scientific  test  process, 
the  OTAs  have  limited  involvement  and  no  control  in  the  development  of  the 
system  models,  nor  in  upgrades  to  the  test  facilities  and  field  ranges,  nor  in 
the  certification  that  the  EC  system  is  ready  for  OT&E.  This  creates  a  major 
challenge  for  OTAs  tasked  with  the  OT&E  of  EC  systems. 


Recommendations 


To  establish  a  disciplined  and  structured  test  process  for  the  acquisition  of 
EC  systems,  I  recommend  the  following. 

1.  The  OTAs  institute  the  scientific  test  process  outlined  in  this 
study.  This  study  highli^ts  the  particular  functions  performed  by  M/S, 
hybrid  GTFs,  and  field  teetuvg  to  overcome  many  of  the  limitations  and  chal¬ 
lenges  associated  with  testing  EC  S3rstems.  It  also  provides  a  scientific  test 
method  that  will  provide  discipline  when  determining  the  operational  effec¬ 
tiveness  and  suitability  of  EC  systems.  Such  a  test  process  will  give  the 
decision  makers  the  confidence  and  satisfaction  that  the  test  results  reflect 
the  way  the  system  will  work  if  deployed.  The  scientific  test  process  will  also 
help  the  OTAs  avoid  the  test-fix-test  scenario  that  began  in  developmental 
testing  by  better  preparing  the  tester  for  the  evaluation  and  by  reporting  the 
results  from  the  evaluation  at  the  conclusion  of  the  test  process.  OT&E 
results  can  then  support  the  intent  of  operational  testing  and  provide  the 
decision  authorities  with  information  to  make  acquisition  decisions.  By  ac¬ 
cepting  this  recommendation  the  OTAs  will  have  a  cost-effective  and  efficient 
method  to  test  EC  systems. 

2.  Provide  education  and  training  on  the  scientific  test  process  for 
EC  ^sterns.  ’The  OTAs  need  to  identify  the  personnel  involved  with  EC 
system  testing  at  all  levels  within  their  organizations  so  that  they  can  receive 
training  on  the  scientific  test  process.  Training  will  faciUtate  the  under¬ 
standing  of  the  EC  test  process  and  show  how  the  results  generated  by  the 
test  process  are  used  by  the  decision  makers  to  support  the  acquisition 
process.  This  training  can  also  reveal  the  purpose  and  application  of  the  T&E 
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tools  used  to  support  the  evaluation  or  assessment  of  EC  systems.  Training 
could  also  show  how  the  test  objectives  are  developed  from  the  COI',  how 
quantitative  and  qualitative  measures  are  developed,  and  where  the  evalua¬ 
tion  criteria  and  data  requirements  come  from.  A  vehicle  already  exists  in 
which  the  necessary  training  could  take  place.  That  vehicle  is  the  OT&E 
training  course  taught  by  AFOTEC  at  Kirtland  AFB,  New  Mexico. 

3.  Clarify  the  requirements  in  APR  55-43,  Management  of  Opera¬ 
tional  Teat  and  Evaluation,  to  have  the  OT&E  final  report  approved 
and  signed  60  days  after  the  last  test  event,  and  clearly  state  that  the 
results  from  OT&E  will  not  be  released  until  the  final  analysis  is 
complete.  To  have  a  disciplined  and  structured  test  process,  suflicient  time 
must  be  allowed  to  complete  the  posttest  evaluation.  The  test  learn  should 
not  be  pressured  into  providing  a  final  report  until  the  posttest  evaluation  is 
complete.  Currently,  the  last  field  test  event  that  gathers  test  data  is  con¬ 
sidered  the  last  test  event  and  according  to  AFR  55-43  a  final  report  is  re¬ 
quired  60  days  later.  This  undoubtedly  will  not  allow  enough  time  to 
complete  any  additional  model  accreditation  efforts  with  field  test  results,  to 
operate  the  models  used  to  determine  mission-level  measures,  to  correlate  the 
test  results  with  the  T&E  tools,  and  to  resolve  any  differences  between  the 
predicted  and  actual  test  results.  In  addition  to  these  final  posttest  activities, 
the  test  team  is  preparing  to  disband  and  ship  the  unit  resources  (computers, 
office  equipment,  software,  test  items  and  equipment,  etc.)  to  the  appropriate 
agencies.  The  result  can  be  a  less  than  thorough  and  complete  anal5r8is  of  the 
test  data.  The  last  test  event  needs  to  be  based  on  the  event  that  completes 
the  posttest  evaluation  phase  and  supports  the  evaluation  of  the  EC  system. 
Of  course  this  event  must  be  documented  in  the  TEMP  and  the  test  plan  so 
that  it  is  clear  when  the  last  event  will  take  place.  As  soon  as  this  event  is 
complete,  the  60-day  countdown  to  the  final  report  can  begin.  This  will  allow 
the  OTA  the  time  needed  to  complete  the  EC  test  process.  It  gives  the  test 
team  the  time  to  complete  the  analysis  of  the  test  results  without  having  to 
rush  into  writing  the  final  report  and  preparing  the  OT&E  briefing.  In  addi¬ 
tion,  it  will  give  the  test  team  a  better  opportunity  to  go  back  through  the 
data  to  verify  trends  just  discovered  or  explain  why  certain  things  happened 
at  the  conclusion  of  testing. 

As  might  be  expected,  the  OTAs  have  a  responsibility  to  report  the  test 
results  on  the  just  completed  phase  of  testing  prior  to  the  milestone  decisions. 
If  the  final  report  is  not  ready  for  distribution,  a  provision  has  been  made  in 
AFR  55-43  that  allows  an  interim  summary  report  in  the  form  of  a  message  to 
suffice.  It  should  summarize  the  evaluation  in  enou^  detail  to  support  the 
decision  makers.  However,  if  the  final  analysis  is  not  complete,  then  the  OTA 
should  not  release  any  interim  findings.  Releasing  the  interim  results  before 
the  posttest  evaluation  is  complete  compromises  the  integrity  of  OT&E  and  is 
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one  of  the  reasons  for  the  perception  that  OT&E  cannot  produce  meaningful 
test  results.  The  schedule  should  not  drive  the  release  of  OT&E  results.  AFR 
55-43  should  clearly  state  that  until  the  final  analysis  is  complete,  the  results 
from  OT&E  will  not  be  released. 

Further  Study 

This  study  does  not  address  the  inadequacies  in  the  T&E  tools  as  they  exist 
today.  These  issues  are  being  worked  throu^  the  various  test  range  and 
facility  improvement  programs.  Another  problem  area  not  addressed  in  this 
study  is  the  attempt  to  develop  a  standardized  set  of  mission-level  measures 
to  use  in  the  evaluation  of  EC  systems.  A  process  needs  to  be  developed  to 
determine  smd  establish  a  set  of  mission-level  MOEs  for  the  evaluation  of  EC 
systems.  This  is  a  topic  broad  enough  to  warrant  a  future  research  study. 
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Abbreviations  and  Acronyms 


ACETEF 

AF 

AFEWES 

AFIA 

AFIC 

AFM 

AFOTEC 

AFR 

AFSC 

ASPJ 

ATIC 


C® 

C^CM 

COEA 

COI 

CSAF 


DemA^al 

DIA 

DMAP 

DOD 

DODD 

DODI 

DOT&E 

DT&E 


A 

Air  Combat  Environment  Test  and  Evaluation 
Facility 
Air  Force 

Air  Force  Electronic  Warfare  Evaluation  Simulator 
Air  Force  Intelligence  Agency 
Air  Force  Intelligence  Command 
Air  Force  Manual 

Air  Force  Operational  Test  and  Evaluation  Center 

Air  Force  Regulation 

Air  Force  Systems  Command 

Airborne  Self-Protection  Jammer 

Avionics  Test  and  Integration  Complex 


c 

Command,  Control,  and  Communications 
Command,  Control,  and  Communications 
Countermeasures 

Cost  and  Operational  Effectiveness  Analysis 
Critical  Operational  Issue 
Chief  of  Staff,  Air  Force 


D 

Demonstration  and  Validation 
Defense  Intelligence  Agency 
Data  Management  and  Analysis  Plan 
Department  of  Defense 
Department  of  Defense  Directive 
Department  of  Defense  Instruction 
Director,  Operational  Test  and  Evaluation 
Developmental  Test  and  Evaluation 


E 


EC 

ECM 

EMC 

EMI 

EMTE 

EGA 

EW 

EWVA 


Electronic  Combat 
Electronic  Countermeasures 
Electnunagnetic  Compatibility 
Electromagnetic  Interference 
Electromagnetic  Test  Environment 
Early  Operational  Assessment 
Electronic  Warfare 

Electronic  Warfare  Vulnerability  Assessment 


F 

FOT&E  Follow-On  Operational  Test  and  Evaluation 

FSD  Full-Scale  Development 


G 


GAO 

General  Accounting  Office 

GPS 

Global  Positioning  S3mtem 

GTF 

Ground  Test  Facility 

GWEF 

Guided  Weapons  Evaluation  Facility 

H 

HITL 

Hardware-in-the-Loop 

HQ 

Headquarters 

IEEE 

1 

Institute  of  Electrical  and  Electronics  Engineers 

IG 

Inspector  General 

IOC 

Initial  Operational  Capability 

lOT&E 

Initial  Operational  Test  and  Evaluation 

IR 

Infrared 

ISD 

Information  Systems  Directive 

ITEA 

International  Test  and  Evaluation  Association 

J 

JCS 

Joint  Chiefs  of  Staff 
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I 


L 

Low-Rate  Initial  Production 

• 

LRIP 

M 

MAJCOM 

Mt^r  Command 

• 

MITL 

Man-in-the-Loop 

MMW 

Millimeter  Wave 

MNS 

Mission  Need  Statement 

MOE 

Measure  of  Effectiveness 

MOP 

Measure  of  Performance 

• 

MS 

Milestone 

M/S 

Modeling  and  Simulation 

N 

• 

NBCC 

Nuclear,  Biological,  and  Chemical  Contamination 

o 

OA 

Operational  Assessment 

• 

ORD 

Operational  Requirements  Document 

OSD 

Office  of  the  Secretary  of  Defense 

OTA 

Operational  Test  Agency 

OT&E 

Operational  Test  and  Evaluation 

• 

P 

PMD 

Program  Management  Directive 

PRIMES 

Preflight  Integration  of  Munitions  and  Electronic 

Systems 

• 

Pub 

Publication 

R 

RCS 

Radar  Cross  Section 

• 

REDCAP 

Real-Time  Electromagnetic  Digitally  ControUed 

Analyzer  and  Processor 

RF 

Radio  Frequency 

R&M 

Reliability  and  Maintainability 
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s 


I 


SAF 

SIL 

SIMVAL 

SOA 

SOF 

SON 

SORD 

SPO 

STAR 

Secretary  of  the  Air  Force 

System  Integration  Laboratory 

Simulator  Validation 

Separate  Operating  Agency 

Special  Operations  Forces 

Statement  of  Operational  Need 

System  Operational  Requirements  Document 

System  Program  Office 

System  Threat  Assessment  Report 

• 

T 

• 

TDD 

T&E 

TEMP 

TPO 

TPWG 

TSPI 

Threat  Description  Document 

Test  and  Evaluation 

Test  and  Evaluation  Master  Plan 

Test  Program  Outline 

Test  Planning  Working  Group 

Time-SpacC'Position  Information 

• 

u 

• 

US 

United  States 

W&A 

V 

Verification,  Validation,  and  Accreditation 

• 
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Terms  and  Definitions 


Attributed  definitions  are  taken  directly  from  sources  cited. 

Accreditation.  The  official  determination  that  a  model  is  acceptable  for  a 
specific  purpose.  (“AFOTEC  Modeling  and  Simulation  Accreditation 
Process,”  4) 

Acquisition.  The  procurement  of  real  property  by  any  means  exclusive  of 
lease  agreements.  The  process  consists  of  planning,  designing,  producing, 
and  distributing  a  system  or  equipment.  (APR  55-43,  Management  of 
Operational  Test  and  Evaluation,  49) 

Acquisition  Program.  A  directed  effort  funded  either  throu^  procurement 
appropriations;  through  the  security  assistance  program;  or  through  the 
research,  development,  test,  and  evaluation  appropriation,  with  the  goal  of 
providing  a  new  or  improved  capability  for  a  validated  need.  An  acquisi¬ 
tion  program  may  include  either  tiie  development  or  procurement  of  sys¬ 
tems,  subsystems,  equipment,  munitions,  or  modifications  to  them,  as  well 
as  supporting  equipment,  systems,  projects,  and  studies.  Excluded  from 
this  definition  .  .  .  are  the  general-purpose,  commercially  available  auto¬ 
matic  data  processing  assets  defined  in  Air  Force  700-8eries  regulations. 
(AFR  SOO-2,  Acquisition  Program  Management,  1) 

Affordability.  A  determination  that  the  life-cycle  cost  of  an  acquisition  pro¬ 
gram  is  in  consonance  with  the  long-range  investment  and  force  structure 
plans  of  the  Department  of  Defense  or  individual  DOD  components.  (DOD 
Instruction  [DODI]  5000.2,  Defense  Acquisition  Management  Policies  and 
Procedures,  15-2) 

Analysis.  The  detailed  examination  and  application  of  disciplined  techniques 
(e.g.,  mathematics  or  statistics)  to  anything  complex  to  understand  its 
nature  or  determine  its  essential  features.  (AFR  55-43,  49) 

Anechoic  Chamber.  A  chamber  in  which  free-space  radiation  of  radio  fre¬ 
quency  (RF)  emissions  can  take  place  that  is  free  from  echoes  and  rever¬ 
berations.  Used  for  testing  an  EC  systems  response  and  performance  to 
RF  stimulation. 

Antenna  Hat.  A  device  placed  over  an  antenna  or  sensor  to  facilitate  the 
exchange  of  sensory  information  to  the  EC  system  or  measurement  in¬ 
strumentation. 

Assess.  To  collect,  analyze,  and  report  data  regarding  an  aspect  of  the  system 
in  test  not  specifically  identified  as  an  operational  requirement.  (AFR 
55-43, 49) 


Atmospheric  Attenuation.  The  process  whereby  the  intensity  of  the  fre¬ 
quency  bands  used  for  radar  is  weakened  as  they  pass  through  the  atmos¬ 
phere.  Attenuation  is  introduced  by  the  air  and  water  vapor,  by  rain  and 
snow,  by  clouds  and  fog,  and  (at  some  frequencies)  by  electrons  in  the 
ionosphere. 

Availability.  A  measure  of  the  degree  to  which  an  item  is  in  au  operable  and 
committable  state  at  the  start  of  a  mission  when  the  mission  is  called  for 
at  an  unknown  (random)  time.  Availability  is  dependent  upon  rehabihty, 
maintainabihty,  and  logistics  supportabiUty.  (AFR  55-43,  49) 

Breadboard.  An  assembly  of  preliminary  circuits  or  components  used  to 
prove  the  feasibility  of  a  device,  circuit,  or  without  regard  to  the 

final  configuration  or  packaging  of  the  parts. 

Capability.  The  ability  to  execute  a  specified  course  of  action.  (Joint  Pub 
1-02,  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms, 
60) 

Combined  Testing.  Testing  conducted  by  the  development  and  operational 
testers  when  due  to  cost,  schedule,  or  test  item  availabUity  they  must 
share  test  facilities  and  resources.  (AFR  80-14,  Test  and  Evaluation,  34) 

Command,  Control,  and  Communications  (C^).  The  process  of,  and  the 
means  for  the  exercise  of  authority  and  direction  by  a  properly  designated 
commander  over  assigned  forces  in  the  accomplishment  of  the 
commsmder’s  mission.  functions  are  performed  through  an  arrange¬ 
ment  of  personnel,  equipment,  communications,  facilities,  and  procedures 
that  are  employed  by  a  commander  in  planning,  directing,  coordinating, 
and  controlling  forces  and  operations  in  the  accomplishment  of  the 
conunander’s  mission.  (AFM  2-8,  Electronic  Combat  (EC)  Operations,  37) 

Compatibility.  Capability  of  two  or  more  items  or  components  of  equipment 
or  material  to  exist  or  function  in  the  same  system  or  environment 
without  mutual  interference.  (Joint  Pub  1-02,  82) 

Computer  Simulations.  Digital  models  of  EC  systems,  host  platforms,  the 
combat  environment  and/or  threat  systems  that  are  executed  together  on 
a  time  and  space  domain  in  a  simulation  of  combat  conditions.  (Air  Force 
Electronic  Combat  Development  Test  Process  Guide  [hereafter  cited  as  AF 
EC  Test  Process  Guide],  B-1) 

Concept  Exploration  and  Definition.  The  identification  and  exploration  of 
alternative  solutions  or  solution  concepts  to  satisfy  a  validated  need, 
usually  through  the  use  of  contracts  with  competent  industry  and  educa¬ 
tional  institutions.  This  phase  requires  the  active  involvement  of  aU  par¬ 
ticipating  commands  to  identify  the  candidate  solutions  and  their 
characteristics.  One  or  more  of  the  selected  candidate  solutions  are  then 
approved  for  entry  into  the  demonstration  and  validation  phase.  [This 
phase  was  formerly  called  conceptual  phase.)  (AFM  11-1,  Air  Force  Glos¬ 
sary  of  Standardized  Terms,  4) 
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Cost  and  Operational  Effectiveness  Analysis  (COEA).  An  analysis  of  the 
estimated  costs  and  operational  effectiveness  of  alternative  materiel  sys¬ 
tems  to  meet  a  mission  need  and  the  associated  program  for  acquiring 
each  alternative.  (DODI  5000.2,  15-3) 

Cost  Threshold.  Expressions  of  value.  They  answer  such  questions  as:  How 
valuable  is  a  given  capability  to  the  service?  How  much  would  the  service 
be  willing  to  give  up  in  order  to  obtain  that  capability?  At  what  point 
would  it  be  preferable  to  drop  the  idea  in  favor  of  some  other  course  of 
action?  (DOD  5000.2-M,  Defense  Acquisition  Management  Documentation 
and  Reports,  8-11) 

Criteria.  Plural  of  criterion. 

Criterion.  A  rule  or  standard  on  which  a  judgment  or  decision  may  be  based. 

Critical  Operational  Issue  (COD.  A  key  operational  effectiveness  or  opera¬ 
tional  suitability  issue  that  must  be  examined  in  operational  test  and 
evaluation  to  determine  the  system’s  capability  to  perform  its  mission.  A 
critical  operational  issue  is  normally  phrased  as  a  question  to  be  answered 
in  evaluating  a  system’s  operational  effectiveness  or  operational 
suitability.  (DODI  5000.2, 15-4) 

Data  Management  and  Analysis  Plan  (DBiAP).  A  plan  that  details  the 
procedures  for  processing  test  data  into  a  form  which  can  be  presented  as 
information  to  support  measures  of  effectiveness  (MOE)  (and  measures  of 
performance  (MOP)].  The  DMAP  should  be  a  useful  guide  for  data  collec¬ 
tion,  reduction,  processing,  and  analysis  for  each  MOE  [and  MOP].  In 
addition,  data  presentation  and  disposal  are  also  included.  If  done  correct¬ 
ly,  analysts  and  data  technicians  who  are  not  familiar  with  the  program 
should  be  able  to  follow  the  steps  outlined  in  [the]  DMAP.  (Operational 
Test  and  Evaluation  Analysts  Handbook,  4-17) 

Data  Processing.  The  preparation  of  source  media  that  contain  data  or  basic 
elements  of  information,  and  the  handling  of  such  data  according  to 
precise  rules  of  procedure  to  accomplish  such  operations  as  cla88if3ring, 
sorting,  calculating,  summarizing,  and  recording.  (AFM  11-1,  25) 

Data  Reduction.  The  action  or  process  of  reducing  data  to  usable  form, 
usually  by  means  of  electron!  computers  and  other  electronic  equipment. 
(AFM  11-1,  25) 

Demonstration  and  Validation  (dem/val).  The  period  when  selected  can¬ 
didate  solutions  are  refined  through  extensive  study  and  analyses: 
hardware  development,  if  appropriate;  tests;  and  evaluations.  The  objec¬ 
tive  is  to  validate  one  or  more  of  the  selected  solutions  and  give  a  basis  for 
deciding  whether  to  proceed  into  full-scale  development.  (AFM  11-1,  4) 

Development.  The  process  of  working  out  and  extending  the  theoretical, 
practical,  and  useful  applications  of  a  basic  design,  idea,  or  scientific  dis¬ 
covery.  The  design,  building,  modification,  or  improvement  of  the 
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prototype  of  a  vehicle,  engine,  instrument,  or  the  like,  as  determined  by 
the  basic  idea  or  concept.  (AFM  11-1,  27) 

Developmental  Teat  and  Evaluation  (DT&E).  That  testing  and  evalua¬ 
tion  used  to  measure  progress;  verify  accomplishment  of  development  ob¬ 
jectives;  and  determine  if  theories,  techniques,  and  materiel  are 
practicable  and  if  systems  or  items  under  development  are  technically 
sound,  reliable,  safe,  and  satisfy  specifications.  (AFM  11-1,  27) 

Digital  Models.  Computer  models  which  consist  entirely  of  software  and 
require  no  unique  hardware  other  than  the  particular  main-frame,  mini, 
or  micro-computer  for  which  they  are  targeted.  (AF  EC  Test  Process  Guide, 
B-2) 

Digital  System  Model  (DSM).  A  computer  model  representing  a  system 
under  development.  The  DSM  is  a  software  equivalent  of  the  system.  It 
aids  engineering  development.  A  DSM  refers  specifically  to  modular 
software  developed  to  run  under  the  common  computer  simulation  ar¬ 
chitecture.  (AF  EC  Test  Process  Guide,  B-2) 

DOD  Components.  The  Office  of  the  Secretary  of  Defense;  the  military 
departments;  the  chairman.  Joint  Chiefs  of  Staff  and  Joint  Staff;  the 
unified  and  specified  commands;  the  defense  agencies;  and  DOD  field  ac¬ 
tivities.  (DODI  5000.2, 15-5) 

Ducting.  Confinement  of  electromagnetic  waves  to  a  restricted  path  in  a 
given  layer  of  air.  A  duct  forms  in  the  troposphere  when  a  layer  of  cool  air 
becomes  trapped  under  a  layer  of  warmer  air,  or  when  a  layer  of  cool  air 
becomes  sandwiched  between  two  layers  of  warmer  air.  The  mode  of 
propagation  between  the  layers  of  air  is  similar  to  that  of  a  waveguide. 
Low-loss  transmission  over  great  distances  is  possible  via  ducts,  very  dis¬ 
tant  radar  echoes  can  be  observed,  and  duct  propagation  constitutes  a 
potential  source  of  interference  between  cochannel  radio  users. 

Early  Operational  Assessment  (E<OA).  An  operational  assessment  con¬ 
ducted  before  or  in  support  of  MS  II.  (AFR  55-43,  51 ) 

EC  Digital  System  ModeL  A  computer  model  representing  the  EC  system. 
This  software  equivalent  of  the  EC  system  is  used  throughout  the  EC  test 
process.  EC  digital  system  models  are  developed  during  the  systems  en¬ 
gineering  process  and  will  model  the  proposed  EC  system  design.  They  are 
used  in  all  levels  of  computer  simulation  to  support  system  development, 
analysis,  and  testing.  The  SPO,  along  with  help  from  the  contractor,  is 
responsible  for  developing  the  EC  digital  system  model.  It  is  also  the  duty 
of  the  SPO  to  maintain  and  update  the  EC  model  throughout  the  acquisi¬ 
tion  process. 

Electromagnetic  Spectrum.  The  range  of  frequencies  of  electromagnetic 
radiation  from  zero  to  infinity.  It  is  divided  into  26  alphabetically  desig¬ 
nated  bands.  (Joint  Pub  1-02, 126) 
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Electronic  Combat  (EC).  Action  taken  in  support  of  military  operations 
against  the  enemy’s  electromagnetic  capabilities.  Electronic  combat  in¬ 
cludes  electronic  warfare  (EW);  elements  of  command,  control,  and  com¬ 
munications  countermeasures  (C^CM);  and  suppression  of  enemy  air 
defenses  (SEAD).  (AFM  11-1,  30) 

Electronic  Combat  (EC)  System.  Any  equipment  used  to  determine,  ex¬ 
ploit,  reduce,  or  prevent  hostile  use  of,  or  retain  friendly  use  of,  the 
electromagnetic  spectrum.  This  includes  associated  support  equipment, 
training  devices,  threat  emitters  and  threat  simulators.  (AFR  55-24, 
Electronic  Warfare  Integrated  Reprogramming  Policy  (U),  2  (Secret)  Infor¬ 
mation  extracted  is  unclassified.  | 

Electronic  Countermeasures  (ECM).  The  division  of  electronic  warfare 
involving  actions  taken  to  prevent  or  reduce  an  enemy^s  effective  use  of 
the  electromagnetic  spectrum  .  .  .  includes  electronic  jamming  and  decep¬ 
tion.  (Joint  Pub  1-02,  127) 

Electronic  Warfare  Vulnerability  Assessment  (EWVA).  The  process  (in¬ 
volving  analysis,  modeling,  simulation,  and  test)  which  determines  the 
degree  of  vulnerability  of  EC  systems  in  their  intended  operational  en¬ 
vironment.  EWVA  is  applied  throughout  the  life  of  a  system  and  includes 
evaluations  of  near-term  (initial  operational  capability),  mid-term  (IOC  + 
5  [years]),  and  far-term  (IOC  +  10  [years])  threats,  including  S3^tem 
specific  countermeasures  to  the  fielded  system.  (AF  EC  Test  Process 
Guide,  B-3) 

Engineering  and  Manufacturing  Development.  The  period  when  the  sys¬ 
tem  and  the  principal  items  necessary  for  its  support  are  designed,  fabri¬ 
cated,  tested,  and  evaluated.  The  intended  output  is,  as  a  minimum,  a 
preproduction  system  that  closely  approximates  the  final  product;  the 
documentation  needed  to  enter  the  production  phase;  and  the  test  results 
that  show  the  product  will  meet  the  requirements.  This  phase  includes  the 
procurement  of  long-lead  pr'^uction  items  and  limited  production  for 
operational  test  and  evaluation.  [This  phase  was  formerly  called  full-scale 
development  (FSD).]  (AFM  11-1, 4) 

Evaluate.  To  collect,  anal3rze,  and  report  against  stated  criteria,  data  about 
an  aspect  of  the  system  in  test  specifically  identified  as  an  operational 
requirement.  (AFR  55-43,  51) 

Evaluation  Criteria.  Stand€U‘ds  used  to  judge  the  achievement  of  required 
operational  effectiveness  or  suitability  characteristics  or  the  resolution  of 
technical  or  operational  issues.  (AFR  80-14,  34) 

Exit  Criteria.  Program  specific  accomplishments  that  must  be  satisfactorily 
demonstrated  before  an  effort  or  program  can  progress  further  in  the 
current  acquisition  phase  or  transition  to  the  next  acquisition  phase.  Exit 
criteria  may  include  such  factors  as  critical  test  issues,  the  attainment  of 
projected  growth  curves  and  baseline  parameters,  and  the  results  of  risk 
reduction  efforts  deemed  critical  to  the  decision  to  proceed  further.  Exit 
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criteria  supplement  minimum  required  accomplishments  and  are  specific 
to  each  acquisition  phase.  (DODI  5000.2,  15-5) 

Extrapolate.  To  project,  extend,  or  expand  known  data  into  an  area  not 
known  or  experienced  so  as  to  gain  some  insight  about  the  unknown  area. 

Feedback.  A  process  by  which  a  system  or  device  is  revised  with  the  output 
from  a  machine,  system,  or  process. 

FoUow-On  Operational  Test  and  Evaluation.  That  test  and  evaluation 
that  is  necessary  during  and  after  the  production  period  to  refine  the 
estimates  made  during  operational  test  and  evaluation,  to  evaluate 
changes,  and  to  reevaluate  the  system  to  ensure  that  it  continues  to  meet 
operational  needs  and  retains  its  effectiveness  in  a  new  environment  or 
against  a  new  threat.  (DODI  5000.2,  15-6) 

Human  Factors.  Those  factors  which  contribute  to  the  optimization  of  a 
system  by  integrating  the  human  performance  necessary  to  operate,  main¬ 
tain,  support,  and  control  the  system  in  its  intended  operational  environ¬ 
ment.  (AFR  80-14,  35) 

Implementiug  Command.  The  command  or  agency  designated  by  Head¬ 
quarters,  United  States  Air  Force  to  manage  an  acquisition  prograun. 
f  AFR  800-2,  1) 

Initial  Operational  Capability  (IOC).  The  first  attainment  of  the 
capability  to  employ  effectively  a  weapon,  item  of  equipment,  or  system  of 
approved  specific  characteristics,  and  which  is  manned  or  operated  by  an 
adequately  trained,  equipped,  and  supported  military  unit  or  force.  (Joint 
Pub  1-02,  181) 

Initial  Operational  Test  and  Evaluation  (lOT&E).  All  operational  test 
and  evaluation  conducted  on  production  or  production  representative  ar¬ 
ticles,  to  support  the  decision  to  proceed  beyond  low-rate  initial  produc¬ 
tion.  It  is  conducted  to  provide  a  valid  estimate  of  expected  system 
operational  effectiveness  and  operational  suitability.  (DODI  5000.2,  15-7) 

Installed  Test  Facility.  Test  resource  which  provides  the  capability  to  test 
EC  systems  while  they  are  installed  on,  or  integrated  with,  host  platforms. 
(AF  EC  Test  Process  Guide,  B-4) 

Instrumentation.  ( 1 )  The  installation  and  use  of  electronic,  g3rroscopic,  and 
other  instruments  for  the  purpose  of  detecting,  measuring,  recording, 
telemetering,  processing,  or  analyzing  different  values  or  quantities  as 
encountered  in  the  flight  of  an  aircraft,  missile,  or  spacecraft.  Instrumen¬ 
tation  applies  to  both  flight-bome  and  ground-based  equipment.  (2)  The 
assemblage  of  such  instruments  in  an  aerospace  vehicle,  with  each  instru¬ 
ment  designed  and  located  so  as  to  occupy  minimum  space,  achieve  mini¬ 
mum  weight,  yet  function  effectively.  (AFM  11-1,  41) 

Intelligence  Report.  A  report  provided  by  the  appropriate  intelligence 
agency/command  to  the  milestone  decision  authority  prior  to  each  mile¬ 
stone  review.  For  milestone  0,  the  report  will  confirm  the  validity  of  the 
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threat  contained  in  the  mission  need  statement.  For  milestones  1-IV,  the 
report  will  confirm  the  validation  of  the  system  threat  assessment  used  in 
support  of  the  program  and  will  address  any  threat  issues  or  unresolved 
threat  concerns  affecting  the  program. 

Interoperability.  The  ability  of  systems,  units  or  forces  to  provide  services  to 
and  accept  services  from  other  systems,  units  or  forces  and  to  use  the 
services  so  exchanged  to  enable  them  to  operate  effectively  together.  (Joint 
Pub  1-02,  190) 

Life-Cycle  Cost.  The  total  cost  to  the  government  of  acquisition  and  owner¬ 
ship  of  that  system  over  its  useful  life.  It  includes  the  cost  of  development, 
acquisition,  support,  and,  where  applicable,  disp>osal.  (DODl  5000.2,  15-9) 

Logistics  Reliability.  A  measure  of  a  system's  ability  to  operate  as  planned 
under  the  defined  operational  and  support  concepts,  using  specified  logis¬ 
tics  resources  (e.g.,  spares  or  manpower).  Logistics  reliability  may  be  ex¬ 
pressed  as  mean  time  between  maintenance,  mean  time  between  removal, 
or  mean  time  between  demand.  It  recognizes  the  effect  of  all  occurrences 
that  place  a  demand  on  the  logistics  support  system  even  when  mission 
capability  is  unaffected.  (AFM  11-1,  45) 

Logistics  Supportability.  The  degree  to  which  planned  logistics  support 
(including  test,  measurement,  and  diagnostic  equipment;  spares  and 
repair  parts;  technical  data;  support  facilities;  transportation  require¬ 
ments;  training;  manpower;  and  software  support)  allows  meeting  system 
avadabiUty  and  wartime  usage  requirements.  (DODI  5000.2, 15-9) 

Low-Rate  Initial  Production  (LRIP).  The  production  of  a  system  in 
limited  quantity  to  provide  articles  for  operational  test  and  evaluation,  to 
establish  an  initial  production  base,  and  to  permit  an  orderly  increase  in 
the  production  rate  sufficient  to  lead  to  full-rate  production  upon  success¬ 
ful  completion  of  operational  testing.  (DODI  5000.2,  15-9) 

Maintainability.  A  measure  of  the  time  or  maintenance  resource  needed  to 
keep  an  item  operating  or  to  restore  it  to  operational  (or  serviceable,  in  the 
case  of  certain  munitions)  status.  Maintainability  may  be  expressed  as  the 
time  to  do  maintenance  (e.g.,  maintenance  downtime  per  sortie),  as  a 
usage  rate  of  manpower  resources  (e.g.,  maintenance — work  hours  per 
fl3ring  hour),  as  the  total  required  manpower  (e.g.,  maintenance  personnel 
per  operational  unit),  or  as  the  time  to  restore  a  system  to  operational 
status  (e.g.,  mean  dovratime).  (AFM  11-1,  46) 

Major  Command  (MAJCOM).  A  major  subdivision  of  the  Air  Force  that  is 
assigned  a  major  part  of  the  Air  Force  mission.  Major  commands  report 
directly  to  Headquarters  United  States  Air  Force  (HQ  USAF).  (AFM  11-1, 
47) 

Many-Versus-Many.  A  test  scenario  in  which  a  friendly  force  consisting  of 
many  aircraft  and  the  EC  system  under  test  is  pitted  against  many  air¬ 
borne  or  ground-based  adversaries. 
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Mature  System.  A  system  is  considered  mature  when  its  reliability  and 
maintainability  characteristics  cease  to  improve  significantly  with  con¬ 
tinued  use.  Systems,  subsystems,  and  components  all  mature  at  various 
rates  for  varying  lengths  of  time.  Unless  otherwise  specified,  a  system  will 
be  considered  to  have  mature  reUability  and  maintainability  (R&M)  char¬ 
acteristics  two  years  after  the  initial  operational  capability  (IOC)  date. 
(AFR  55-43,  54) 

Measure.  Any  standard  of  comparison,  estimation,  or  judgment;  criterion. 
(The  World  Book  Dictionary,  1287) 

Measurement  Facilities.  Test  resources  used  for  exploring  and  evaluating 
EC  technologies.  Data  collected  from  these  resources  include  antenna  pat¬ 
terns,  radar  cross  sections,  and  infrared  and  laser  signatures.  (AF  EC  Test 
Process  Guide,  B-5) 

Measure  of  Effectiveness  (MOE).  A  qualitative  or  quantitative  measure  of 
a  system’s  performance  at  a  mission  level,  or  a  characteristic  that  indi¬ 
cates  the  degree  to  which  the  system  performs  the  task  or  meets  the 
mission-level  requirement  under  specified  conditions. 

Measure  of  Performance  (MOP).  A  quahtative  or  quantitative  measure  of 
a  system’s  performance  at  a  system  level,  or  a  characteristic  that  indicates 
the  degree  to  which  the  system  performs  the  task  or  meets  the  system- 
level  requirement  under  specified  conditions. 

Milestones  (MS).  Major  decision  points  that  separate  the  phases  of  an  ac¬ 
quisition  program.  (DOD  Directive  5000.1,  Defense  Acquisition,  2) 

Minimum  Acceptable  Operational  Requirement.  The  value  for  a  par¬ 
ticular  parameter  that  is  required  to  provide  a  system  capability  that  will 
satisfy  the  validated  mission  need.  Also  known  as  the  performance 
threshold.  (DODI  5000.2, 15-11) 

Mission  Need  Statement  (MNS).  A  document  prepared  by  the  respective 
using  command  or  HQ  USAF  that  identifies  an  operational  deficiency  that 
cannot  be  satisfied  through  changes  in  tactics,  strategies,  doctrine,  or 
training.  A  solution  normally  entails  research  and  development,  produc¬ 
tion,  and  proG\irement  of  a  new  capability  or  modification  of  an  existing 
system.  [Formerly  called  Statement  of  Operational  Need  (SON).]  (AFM 
11-1,72) 

Mission  Reliability.  A  measure  of  the  ability  of  a  system  to  complete  its 
planned  mission  or  function.  Mission  reliability  may  be  expressed  as  mis¬ 
sion  completion  success  probability,  mean  mission  duration,  or  as  mean 
time  between  critical  failure,  as  appropriate.  (AFM  11-1,  50) 

ModeL  A  representation  of  an  actual  or  conceptual  system  that  involves 
mathematics,  logical  expressions,  or  computer  simulations  that  can  be 
used  to  predict  how  the  system  mi^t  perform  or  survive  under  various 
conditions  or  in  a  range  of  hostile  environments.  (DODI  5000.2,  15-11) 


Multispectral  Threat  Environment.  An  environment  consisting  of  threat 
emissions  from  several  spectra,  especially  from  parts  of  the  visible,  in¬ 
frared,  and  microwave  spectra. 

One-Versus-Few  Engagement.  A  scenario  in  which  a  single  EC  system 
under  test  is  pitted  against  several  adversaries. 

One-Versus-One  Engagement.  A  scenario  in  which  a  single  EC  system 
under  test  is  pitted  against  a  single  adversary. 

Operational  Assessment  (OA).  An  evaluation  of  operational  effectiveness 
and  operational  suitability  made  by  an  independent  operational  test  ac¬ 
tivity,  with  user  support  as  requued,  on  other  than  production  systems 
The  focus  of  an  operational  assessment  is  on  significant  trends  noted  in 
development  efforts,  programmatic  voids,  areas  of  risk,  adequacy  of  re¬ 
quirements,  and  the  ability  of  the  program  to  support  adequate  opera¬ 
tional  testing.  Operational  assessments  may  be  made  at  any  time  using 
technology  demonstrators,  prototypes,  mockups,  engineering  development 
models,  or  simulations  but  will  not  substitute  for  the  independent  opera¬ 
tional  test  and  evaluation  necessary  to  support  full  production  decisions 
(DODI  5000.2,  15-13) 

Operational  Effectiveness.  The  overall  degree  of  mission  accomplishment 
of  a  system  when  used  by  representative  personnel  in  the  environment 
planned  or  expected  (e.g.,  natural,  electronic,  threat,  etc.)  for  operational 
emplo3rment  of  the  system  considering  organization,  doctrine,  tactics,  sur¬ 
vivability,  vulnerabihty,  and  threat  (including  countermeasures,  initial 
nuclear  weapons  effects,  nuclear,  biological,  and  chemical  contamination 
(NBCC)  threats).  (DODI  5000.2, 15-13) 

Operational  Reliability.  The  probability  that  au  operationally  ready  system 
will  react  as  required  to  accomplish  its  intended  mission  or  function  as 
planned,  excluding  the  effects  of  enemy  action;  may  be  specified  as  an 
estimated  or  an  achieved  rehability.  (AFR  55-43,  55) 

Operational  Requirement.  An  established  need  justifying  the  timely  al¬ 
location  of  resources  to  achieve  a  capability  to  accomplish  approved 
military  objectives,  missions,  or  tasks.  (Joint  Pub  1-02,  264) 

Operational  Requirements  Document  (ORD).  A  document  prepared  by 
the  respective  using  command  that  describes  pertinent  quantitative  and 
qualitative  performance,  operation,  and  support  parameters,  charac¬ 
teristics,  and  requirements  for  a  specific  candidate  weapon  system.  The 
[ORD]  documents  how  a  83rstem  will  be  operated,  deployed,  employed,  and 
supported  and  provides  initial  guidance  for  the  implementing,  supporting, 
and  participating  commands  and  agencies.  [Formerly  called  System 
Operational  Requirements  Document  (SORD).]  (AFM  11-1,  74) 

Operational  Suitability.  The  degree  to  which  a  system  can  be  placed  satis¬ 
factorily  in  field  use  with  consideration  given  to  availabilify,  compatibility, 
transportability,  interoperability,  reliability,  wartime  usage  rates,  main- 


89 


tainability,  safety,  human  factors,  manpower  supportabiiity,  logistics  sup- 
portability,  natural  environmental  effects  and  impacts,  documentation, 
and  training  requirements.  (IX)DI  5000.2, 15-13) 

Operational  Test  Agency  (OTA).  The  command  or  agency  designated  by 
the  program  management  directive  (PMD)  or  other  appropriate  program 
directive  as  responsible  for  managing  the  independent  OT&E  of  a  system. 
(APR  55-43,  55) 

Operational  Test  and  Evaluation  (OT&E).  Testing  and  evaluation 
(divided  into  initial  o{)erational  test  and  evaluation  and  follow-on  opera¬ 
tional  test  and  evaluation,  and  generally  associated  with  the  first  major 
production  decision)  conducted  in  as  realistic  an  operational  environment 
as  possible  to  estimate  the  prospective  system’s  military  utility,  opera¬ 
tional  effectiveness,  and  operational  suitability.  In  addition,  operational 
test  and  evaluation  provides  information  on  organization,  personnel  re¬ 
quirements,  doctrine,  and  tactics.  Also,  it  may  provide  data  to  support  or 
verify  material  in  operating  instructions,  publications,  and  handbooks. 
(AFM  11-1,  54) 

Operational  Test  and  Evaluation  (OT&E)  Plan.  The  document  prepared 
by  the  OT&E  agency  that  details  the  background,  scope,  and  procedures 
for  conducting  OT&E.  (AFR  55-43,  55) 

Participating  Command.  A  command  or  agency  designated  by  Head¬ 
quarters  United  States  Air  Force  to  support  and  advise  the  program 
manager  (PM).  The  supporting  command  is  also  a  participating  command. 
(AFR  800-2,  2) 

Performance.  Those  operational  and  support  characteristics  of  the  system 
that  allow  it  to  effectively  and  efficiently  perform  its  assigned  mission  over 
time.  The  support  characteristics  of  the  system  include  both  supportabiiity 
aspects  of  the  design  and  the  support  elements  necessary  for  system 
operation.  (DODI  5000.2,  15-13) 

Performance  Threshold.  See  minimum  acceptable  operational  require¬ 
ment. 

Preproduction-Representative.  An  article  in  final  form  empIo}dng  stan¬ 
dard  parts  representative  of  articles  to  be  produced  on  a  production  line 
with  production  tooling. 

Production  and  Deployment.  The  period  from  production  approval  until 
the  last  system  is  delivered  and  accepted.  The  objective  is  to  efficiently 
produce  and  deliver  effective  and  supportable  systems  to  the  operating 
units.  This  includes  the  production  of  all  principal  and  support  equipment. 
The  deployment  phase  encompasses  the  process  of  uniting  facilities, 
hardware  and  software,  personnel,  and  procedural  publications;  and 
delivering  an  acceptable  integrated  system  to  the  using  and  supporting 
commands.  This  overlaps  the  production  phase.  [This  phase  was  formerly 
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divided  into  two  separate  phases  called  production  phase  and  deployment 
phase.]  (AFM  11-1, 4) 

Production  System.  A  system  which  is  in  final  form,  employs  standard 
parts,  and  is  representative  of  final  equipment. 

Program  Management  Directive  (PMD).  The  official  Air  Force  document 
used  to  direct  acquisition  or  modification  responsibilities  to  appropriate 
Air  Force  MAJCOMs  for  the  development,  acquisition,  or  modification  of  a 
specific  weapon  system,  subsystem,  or  piece  of  equipment.  It  is  used 
throughout  the  acquisition  cycle  to  terminate,  initiate,  or  direct  research; 
development;  production;  or  Class  Ill,  IV,  or  V  modifications  for  which 
sufficient  resources  have  been  identified.  States  program  unique  require¬ 
ments,  goals,  and  objectives,  especially  those  to  be  met  at  each  acquisition 
milestone  or  program  review.  (AFM  11-1,  59) 

Protot3rpe.  A  model  suitable  for  evaluation  of  design,  performance,  and 
production  potential.  (Joint  Pub  1-02, 290) 

Realistic  Test  Ehivironment.  The  conditions  under  which  the  system  is 
expected  to  be  operated  and  maintained,  including  the  natural  weather 
and  climatic  conditions,  terrain  effects,  battlefield  disturbances,  and 
enemy  threat  conditions.  (AFR  55-43,  57) 

Reliability.  The  ability  of  a  system  and  its  parts  to  perform  its  mission 
without  failiire,  degradation,  or  demand  on  the  support  system. 

Research  Testing.  Operations  performed  as  a  part  of  research  experiments 
and  investigations  to  measure,  verify,  or  assess  phenomena,  hypotheses, 
and  resxilts  of  experimentation;  and  to  gain  new  knowledge.  (AFM  11-1, 
66) 

Simulation.  A  method  for  implementing  a  model.  It  is  the  process  of  conduct¬ 
ing  experiments  with  a  model  for  the  purpose  of  understanding  the  be¬ 
havior  of  the  system  modeled  under  selected  conditions  or  of  evaluating 
various  strategies  for  the  operation  of  the  system  within  the  limits  im¬ 
posed  by  developmental  or  operational  criteria.  Simulation  may  include 
the  use  of  analog  or  digital  devices,  laboratory  models,  or  “testbed”  sites. 
Simulations  are  usually  programmed  for  solution  on  a  computer;  however, 
in  the  broadest  sense,  mUluiry  exercises  and  wargames  are  also  simula¬ 
tions.  (DODI  5000.2,  15-15) 

Simulator.  A  generic  term  used  to  describe  a  family  of  equipment  used  to 
represent  threat  weapon  systems  in  developmental  testing,  operational 
testing,  and  training.  A  threat  simulator  has  one  or  more  characteristics 
which,  when  detected  by  human  senses  or  man-made  sensors,  provide  the 
appearance  of  an  actual  threat  weapon  system  with  a  prescribed  degree  of 
fideUty.  (DODI  5000.2, 15-15) 

Supportability.  The  degree  to  which  system  design  characteristics  and 
planned  logistics  resources,  including  manpower,  meet  system  peacetime 
readiness  and  wartime  utilization  requirements.  (DODI  5000.2, 15-16) 
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Supporting  Command.  The  command  assigned  responsibility  for  providing 
logistics  support;  it  assumes  program  management  responsibility  from  the 
implementing  command.  ( AFR  800-2,  2) 

System  Effectiveness.  A  measure  of  the  extent  to  which  a  system  may  be 
expected  to  achieve  a  set  of  specific  mission  requirements  expressed  as  a 
function  of  availability,  dependability,  and  capability.  (AFM  11-1,  74) 

System  Program  Office  (SPO).  The  organization  comprised  of  technical, 
administrative,  and  business  management  personnel  assigned  full  time  to 
a  system  program  director.  The  office  may  be  augmented  with  additional 
personnel  from  participating  organizations.  (AFM  11-1,  74) 

System  Threat  Assessment.  Describes  the  threat  to  be  countered  and  the 
projected  threat  environment.  The  threat  information  should  reference 
DIA  or  Service  Technical  Intelligence  Center  approved  dc«.ument»  (1)01)1 
5000.2,  15-17) 

System  Threat  Assessment  Report  (STAR).  An  intelligence  document 
that  serves  as  the  single  authoritative  reference  for  threat  data  regardioR 
a  weapon  system  acquisition  program.  It  describes  the  lethal  and  non- 
lethal  threats  against  the  proposed  US  system  and  the  threat  environ¬ 
ment  in  which  the  system  will  operate.  (AFM  11-1,  75) 

Test  and  Evaluation  (T&E).  Actions  taken  to  ensure  that  a  system  achieves 
performance  requirements.  The  term  test  denotes  any  project  or  program 
designed  to  obtain,  verify,  and  provide  data  to  evaluate  research  and 
development  other  than  laboratory  experiments;  progress  in  accompbsh- 
ing  development  objectives;  performance  and  operational  capability  of  sys¬ 
tems,  subsystems,  and  components;  and  equipment  items.  The  term 
evaluation  denotes  the  review  and  analysis  of  quantitative  data  produced 
during  current  or  previous  testing  and  data  obtained  from  tests  conducted 
by  other  government  agencies  and  contractors,  from  operation  and  com¬ 
mercial  experience,  or  combinations  thereof.  (AFR  55-43,  59) 

Test  and  Evaluation  Master  Plan  (TEMP).  The  basic  planning  document 
for  all  T&E  related  to  a  particular  system  acquisition,  and  is  used  in 
planning,  reviewing  and  approving  T&E.  It  is  required  for  all  major 
defense  acquisition  programs,  all  OSD  oversight  programs,  all  HQ  USAF 
programs  directed  by  a  PMD,  and  may  be  required  for  an  ISD  directed 
information  system  program.  The  TEMP  integrates  critical  issues,  test 
objectives,  evaluation  criteria,  system  characteristics,  responsibilities, 
resources,  and  schedules  for  T&E.  (AFR  55-43,  59) 

Test  Article.  The  EC  system  or  system  components  to  be  tested. 

Testbed.  A  system  representation  consisting  partially  of  actual  hardware 
and/or  software  and  partially  of  computer  models  or  prototype  hardware 
and/or  software.  (DODI  5000.2, 15-17) 
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Test  Conditions.  The  environment  (eg.,  location,  weather),  scenario,  and 
operating  procedures  and  configurations  for  the  SLIT  |  system  under  test  I 
and  adversaries  in  the  i  scenario.  {AF  EC  Test  Process  Guide,  B-9) 

Test  Design.  The  com'.mationis)  of  test  trial,  test  environment,  and  test 
condition  parameters  used  to  construct  a  test  sequence.  (AF  EC  Test 
Process  Guide,  B-9) 

Test  Director.  The  person  assigned  to  conduct  a  test  according  to  a  test  plan, 
wh<.  exercises  overall  responsibility  for  achieving  plan  objectives.  (AFR 
r  ,-43,  59) 

Test  Environment.  The  test  location,  atmospheric  conditions,  multispectral 
emissions  from  the  threat  emitters  or  test  items,  instrumentation  and 
collection  devices,  and  so  forth,  in  which  the  test  is  conducted. 

Test  Event.  An  activity  during  conduct  of  a  test  trial  that  requires  a  respon.se 
by  the  system  and/or  personnel  under  test  {AF  EC  Test  Process  Guide. 
B-9) 

Test  Manager.  The  person  designated  as  the  focal  point  for  advance  plan¬ 
ning,  OT&E  planning,  executing  the  test,  and  reporting  on  an  OT&E 
program.  (AFR  55-43,  59) 

Test  Objective.  The  specific  performance  or  technical  parameters  to  be 
measured  during  the  test  to  evaluate  system  performance,  system  opera¬ 
tional  effectiveness,  or  system  suitability.  {AF EC  Test  Process  Guide,  B-9) 

Test  Planning  Working  Group  (TPWG).  I A  team  |  assigned  by  the  program 
manager  to  provide  a  forum  for  test-related  subjects;  assists  in  estab¬ 
lishing  test  objectives  and  evaluation  baselines;  defines  organization, 
responsibilities,  and  relationships;  estimates  costs  and  schedules;  and 
identifies  needed  test  resources  Normally,  includes  system  program  office 
(SPO)  representatives,  AFSC  test  agencies,  contractors,  AFOTEC,  and 
using  and  supporting  major  commands.  (AFR  55-43,  59) 

Test  Program  Outline  (TPO).  The  basic  resource  management  document 
used  throughout  the  OT&E  planning  process.  It  identifies  resources  re¬ 
quired  to  support  testing  and  is  the  basis  for  budget  submissions,  man¬ 
power  plans,  and  procurement  leadtime.  (AFR  55-43,  59) 

Test  Resources.  Assets  that  support  the  test  and  evaluation  of  the  EC  sys¬ 
tem.  They  include  the  test  facilities,  models,  threat  simulators,  friendly 
support  assets,  instrumentation  systems,  range-tracking  systems,  mission 
control  centers,  test  facilities,  environment  generators,  and  so  forth. 

Test  Scenario.  A  situation,  representative  of  what  the  system  under  test 
may  encounter  in  real  life,  that  is  used  to  enact  a  set  of  events  between  it 
and  adversaries  included  in  the  situation  (e.g.,  threat  simulator  locations 
and  flight  profiles).  {AF  EC  Test  Process  Guide,  B-9) 

Test  Sequence.  The  ordering  from  first  to  last  of  the  test  trials  in  the  test 
design.  (AF  EC  Test  Process  Guide,  B-9) 
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Test  Trial.  A  single  case  of  a  test  design  that  measures  the  values  of  the 
performance  or  technical  parameters  in  a  specific  test  facility  under  the 
same  test  conditions.  (AF  EC  Test  Process  Guide,  B-9) 

Threat  Replica.  A  device  which  has  all  the  externally  observable  charac¬ 
teristics  of  and  is  a  close  reproduction  of  the  actual  threat  system. 

Threshold.  Minimum  acceptable  value  for  a  performance  parameter  neces¬ 
sary  to  provide  an  operational  capability  that  will  satisfy  the  mission  need. 
(AFR  57-1,  Air  Force  Mission  Needs  and  Operational  Requirements 
Process,  133) 

Time-Space-Position  Information  (TSPI).  A  coordinate  reference  system 
that  provides  a  method  to  locate  an  object  in  space  and  time.  It  can  consist 
of  several  types  of  coordinate  reference  systems  such  as  global  positioning 
system  (GPS),  radar,  laser  range  tracking  cinesextant,  cinetheodolite, 
high-speed  camera,  and  so  forth.  A  TSPI  system  is  used  to  track  aircraft  or 
missiles;  to  provide  overall  mission  control;  to  determine  the  three-dimen¬ 
sional  position  of  the  system  under  test;  to  provide  positive  identification 
of  aircraft  on  the  range  for  evaluating  mission  or  engagement  events;  and 
to  determine  aircraft  or  missile  velocity,  acceleration,  and  attitude  for 
inputs  to  scoring  simulations. 

Uninstalled  Test  Facility.  A  test  resource  which  provides  the  capability  to 
test  EC  systems  before  they  are  installed  or  mounted  in  their  host  plat¬ 
form  for  the  purpose  of  evaluating  the  effectiveness  of  the  EC  systems 
hardware  and  techniques. 

Using  Command.  The  major  command  responsible  for  system  employment. 
(AF  EC  Test  Process  Guide,  B-9) 

Validation.  The  process  of  determining  the  degree  to  which  a  model  is  an 
accurate  representation  of  the  real  world  from  the  perspective  of  the  in¬ 
tended  uses  of  the  model.  Validation  is  also  a  process  of  building  con¬ 
fidence  in  the  model  to  a  satisfactory  level  so  that  a  valid  inference  about 
the  actual  system  can  be  made  with  the  model.  (“AFOTEC  Modeling  and 
Simulation  Accreditation  Process,”  3) 

Verification.  The  process  of  determining  that  a  model  implementation  ac¬ 
curately  represents  the  developer’s  conceptual  description  and  specifica¬ 
tion.  It  is  a  process  that  is  used  to  ensure  a  model’s  internal  data, 
structure  and  logic  correctly  behaves  in  a  manner  that  represents  the 
system  being  modeled.  (“AFOTEC  Modeling  and  Simulation  Accreditation 
Process,”  2) 

Waveguide.  A  device  (such  as  a  duct,  coaxial  cable  or  glass  fiber)  designed  to 
confine  and  direct  the  propagation  of  electromagnetic  waves. 
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